An animal behavior recognition method based on multi-scale feature fusion

Abstract

The application relates to the field of animal behavior monitoring, and specifically discloses an animal behavior recognition method based on multi-scale feature fusion, which comprises the following steps: S1, multi-modal data synchronous acquisition and preprocessing; S2, multi-scale visual feature extraction; S3, audio feature extraction and coding; S4, cross-modal feature fusion; S5, classifier calibration and behavior recognition; S6, model training and optimization. The recognition method enhances the robustness of behavior recognition through multi-modal (vision + sound) and multi-scale feature fusion; the classifier calibration mechanism is adopted to relieve the class imbalance problem; and the overall method has higher accuracy and practicability in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of animal behavior monitoring, and in particular to an animal behavior recognition method based on multi-scale feature fusion. Background Technology

[0002] In the field of wildlife conservation and research, accurate identification of abnormal animal behavior is crucial for assessing population health, providing early warning of disease outbreaks, and monitoring environmental stress. Traditional field observation methods are not only time-consuming and labor-intensive but also susceptible to subjective interference. With technological advancements, automated monitoring solutions based on computer vision are gradually being applied.

[0003] However, existing technical solutions have significant limitations. First, most methods heavily rely on visual information collected by static cameras deployed at fixed locations, resulting in single-modal analysis. In complex natural environments, vegetation obstruction, poor lighting conditions, and darkness at night are frequent occurrences, easily leading to missing or degraded visual information and creating blind spots in identification. Second, existing methods typically analyze "abnormal movement" (such as wandering into uninhabited areas) separately from "abnormal individual behavior" (such as stereotyped behavior), failing to establish a correlation between the two. For example, simply tracking population movement anomalies cannot simultaneously and accurately determine which specific individual in the group exhibited what kind of behavioral abnormality and its potential causes (such as injury or fright), leaving the analysis superficial and limiting its early warning value. Furthermore, wildlife behavior data suffers from severe class imbalance, with normal behavior samples making up the vast majority while valuable abnormal behavior samples are scarce. This makes data-driven identification models highly susceptible to bias and lacks sensitivity in identifying rare but crucial abnormal behaviors (such as early pathological conditions).

[0004] Therefore, there is an urgent need in this field for a new intelligent analysis method that can integrate multi-source information, collaboratively analyze population and individual anomalies, and effectively overcome the problem of data imbalance, so as to break through the existing technical bottlenecks and provide more timely and accurate decision support for wildlife conservation. Summary of the Invention

[0005] To address the shortcomings of existing animal behavior recognition technologies, which primarily rely on single image modalities and suffer from insufficient recognition accuracy in scenarios with occlusion, varying lighting, or complex backgrounds, and the fact that normal behavior samples far outnumber abnormal behavior samples in animal behavior data, leading to low recognition rates for rare abnormal behaviors due to class imbalance, this invention provides an animal behavior recognition method based on multi-scale feature fusion.

[0006] The technical solution adopted in this invention is:

[0007] An animal behavior recognition method based on multi-scale feature fusion includes the following steps:

[0008] S1. Multimodal Data Synchronous Acquisition and Preprocessing: Simultaneously acquire video stream data and audio stream data containing individual animals through infrared cameras and microphone arrays deployed in the monitoring area; perform animal target detection and cross-frame tracking on the video stream data, and perform noise reduction processing on the audio stream data;

[0009] S2. Multi-scale visual feature extraction: For each tracked animal individual, extract sub-segments of different scales from its video clips, including complete individual video clips at normal frame rate, video clips processed by reducing frame rate, and local video clips centered on key parts, and extract the corresponding visual feature vectors for each.

[0010] S3. Audio Feature Extraction and Encoding: Calculate Mel spectrograms for segments of the audio stream data and extract audio feature vectors;

[0011] S4. Cross-modal feature fusion: After fusing multi-scale visual feature vectors, they are fused with audio feature vectors through an attention mechanism to obtain a fused feature vector;

[0012] S5. Classifier Calibration and Behavior Recognition: Input the fused feature vector into the calibration module, which contains a fixed-parameter classifier and a learnable-parameter classifier. Obtain the final behavior category probability distribution through the weighted fusion classification result.

[0013] S6. Model Training and Optimization: Use a class-balanced loss function to train the overall model end-to-end.

[0014] Preferably, in S1, the individual animal target detection uses the YOLOv8n model to obtain a bounding box sequence. ,in For the temporal index of video frames, The total number of video frames. , , , , They represent the first The frame's bounding box's center x-coordinate, center y-coordinate, width, and scheduling; the cross-frame tracking uses the DeepSORT algorithm to output the tracking trajectory. ,in, For individual identification, The time interval for the trajectory.

[0015] Preferably, in S2, the specific process of multi-scale visual feature extraction includes: extracting the first sub-segment. This is a regular frame rate video clip, the second sub-segment. This is the third sub-segment of the video after frame rate reduction processing. For key parts of the video clips, feature vectors are extracted using three independent ResNet-50 networks. ;in, For video space, , , These are the frame numbers of the extracted first, second, and third sub-segments, respectively. , The height of the video frame. The width of the video frame. The height of a local video segment frame. The width of the local video segment frame, and , , To expand the scaling parameter, 3 represents the number of channels in the RGB color image. is the dimension of the visual feature vector.

[0016] Preferably, the center point of the key part of the third sub-segment is located using the OpenPose pose estimation algorithm, and the key point position is determined. ,in, , These are the x and y coordinates of the key points in the image, respectively. , For two-dimensional coordinate indexes on a heatmap, For the first Heatmaps of key points, with local areas expanding outwards from these key points at a ratio of [missing information]. .

[0017] Preferably, in S3, the Mel spectrogram calculation includes: pre-emphasizing the audio segment, windowing the frame, performing a short-time Fourier transform to obtain the amplitude spectrum, and then passing it through a Mel filter bank. The mapping calculation formula is:

[0018] ;

[0019] in, This is the time frame index for the audio spectrogram. Add an index to Mel. For frequency index, For frequency points, For the first in the Mel filter bank Each filter at frequency The weight of the position, For short-time Fourier transform in time and frequency Complex values ​​at that location, For Mel spectrogram in time and Mel bring The energy value at that location, To take the modulus of the complex number.

[0020] Preferably, in S4, the cross-modal fusion employs a multi-head attention mechanism, specifically including: concatenating multi-scale visual features and then reducing their dimensionality to obtain... Then calculate the output of each attention head:

[0021] ;

[0022] in, The fused visual feature vector To integrate the dimensions of features, For the first The output of each attention head, For the first A query matrix with attention heads For the first The key matrix of each attention head. for The transpose of the matrix, For the first The value matrix of each attention head, For audio feature vectors, For the projection dimension, For the total number of attention heads, For learnable weight matrix, For the first The query projection weight matrix of each attention head For the first The key projection weight matrix of each attention head. For the first The values ​​of each attention head are projected onto the weight matrix. This is the scaling factor;

[0023] The outputs of all heads are concatenated and passed through a linear layer to obtain a fused feature vector.

[0024] Preferably, in S5, the classifier calibration module includes an isoangular tight-frame classifier. and learnable classifiers ,in Satisfy mathematical constraints:

[0025] ;

[0026] in, The weight matrix for the isoangular tight frame classifier is for The transpose of the matrix, For the learnable classifier weight matrix, The total number of behavior categories. for 3D identity matrix for A dimensional vector of all 1s for The transpose of ;

[0027] Final behavior category probability distribution The calculation is as follows:

[0028] , , ;

[0029] in, For learnable equilibrium parameters, The output of the logical value for the isoangular tight frame classifier. The output of the learnable classifier is the logical value. To fuse feature vectors, for The transpose of .

[0030] Preferably, in S6, the loss function is a class-balanced focus loss, for a sample The loss is calculated as follows:

[0031] , ;

[0032] in, For training sample index, For the first The true class label of each sample To balance the weights for each category, For the prediction sample Category The probability, For aggregation parameters, For categories in the training set The number of samples, This is a hyperparameter.

[0033] Preferably, after S4, a temporal feature modeling step is included: fusing the feature vectors Organized into a sequence according to time. Input the bidirectional long short-term memory network and calculate the hidden states:

[0034] ;

[0035] in, For time step index, This represents the total number of time steps. For time steps The fused feature vector at that location, For Long Short-Term Memory Networks at time steps The hidden state at that location For Long Short-Term Memory Networks at time steps The hidden state at that location This is the parameter set for Long Short-Term Memory (LSTM) networks. The computational function for the Long Short-Term Memory (LSTM) network, and the hidden state at the last time step. As an enhanced temporal feature vector, it is used for classifier calibration in S5.

[0036] Preferably, after S5, a behavioral interpretability analysis step is also included:

[0037] Based on the attention weights of S4 and the classifier weights of S5, the feature contribution is calculated; for the identified abnormal behaviors, the key feature regions and important sound segments are visualized, and text descriptions explaining the causes of the behaviors are generated.

[0038] The beneficial effects of this invention are:

[0039] By fusing multimodal (visual + auditory) and multi-scale features, the robustness of behavior recognition is enhanced; a classifier calibration mechanism is adopted to alleviate the class imbalance problem; the overall method has higher accuracy and practicality in complex scenarios. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the identification method in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the system architecture corresponding to the identification method in this embodiment of the invention;

[0042] Figure 3 This is a schematic diagram of multi-scale visual feature extraction in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of a cross-modal feature fusion mechanism based on multi-head attention in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram illustrating the working principle of the classifier calibration module in an embodiment of the present invention. Detailed Implementation

[0045] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0046] Example

[0047] like Figure 1 , Figure 2 As shown, an animal behavior recognition method based on multi-scale feature fusion includes the following steps:

[0048] S1. Multimodal Data Synchronous Acquisition and Preprocessing: Simultaneously acquire video stream data and audio stream data containing individual animals through infrared cameras and microphone arrays deployed in the monitoring area; perform animal target detection and cross-frame tracking on the video stream data, and perform noise reduction processing on the audio stream data;

[0049] S2. Multi-scale visual feature extraction: For each tracked animal individual, extract sub-segments of different scales from its video clips, including complete individual video clips at normal frame rate, video clips processed by reducing frame rate, and local video clips centered on key parts, and extract the corresponding visual feature vectors for each.

[0050] S3. Audio Feature Extraction and Encoding: Calculate Mel spectrograms for segments of the audio stream data and extract audio feature vectors;

[0051] S4. Cross-modal feature fusion: After fusing multi-scale visual feature vectors, they are fused with audio feature vectors through an attention mechanism to obtain a fused feature vector;

[0052] S5. Classifier Calibration and Behavior Recognition: Input the fused feature vector into the calibration module, which contains a fixed-parameter classifier and a learnable-parameter classifier. Obtain the final behavior category probability distribution through the weighted fusion classification result.

[0053] S6. Model Training and Optimization: Use a class-balanced loss function to train the overall model end-to-end.

[0054] Visual and acoustic modalities are complementary in terms of information. The visual modality can capture spatial details (such as limb movements) but is susceptible to occlusion or lighting conditions; the acoustic modality provides temporal context (such as vocalization frequency) and is not limited by visual occlusion. Through synchronous acquisition, the system can use acoustic information to compensate for visual deficiencies. For example, when an animal is occluded, acoustic features can still indicate the type of behavior (such as painful vocalizations corresponding to abnormal behavior), thereby enhancing reliability at the data source level.

[0055] Different animal behaviors may manifest at different spatiotemporal scales. Global video clips (at a normal frame rate, typically the frame rate at the time of recording, such as 30fps) capture macroscopic behavioral patterns (e.g., running), clips with reduced frame rates (e.g., down to 5fps) highlight slow or sustained behaviors (e.g., pacing), and local clips focus on details of key areas (e.g., ear twitching). This multi-scale design ensures that the model can capture both global and local features simultaneously, avoiding feature omissions caused by a single scale. For example, local clips can amplify subtle movements, increasing sensitivity to mildly abnormal behaviors.

[0056] Attention mechanisms can adaptively weight the feature contributions between modalities, using visual features as queries and sound features as keys / values ​​to calculate relevance weights. This allows the model to dynamically highlight modal information related to the current behavior (e.g., in an attack, the weights of visual actions and roaring sounds increase simultaneously), rather than simply splicing features, thereby reducing noise interference and improving the representational ability of fused features.

[0057] Fixed-parameter classifiers enforce mathematical constraints on the decision boundary, and learnable classifiers adapt to data distributions. Weighted fusion balances prior knowledge (theoretical optimal boundary) and data-driven knowledge (actual distribution), preventing the model from being biased towards the majority class due to imbalanced training data. For example, rare anomalous behaviors can still obtain fair classification probabilities with a small number of samples.

[0058] The loss function adjusts the training gradient through weight terms and focusing parameters, making the model pay more attention to minority class samples and hard samples. This alleviates the class imbalance problem from an optimization perspective and ensures that the model has a good uniform recognition ability for all behavior categories.

[0059] The above steps significantly improve the robustness and accuracy of behavior recognition by fusing multimodal (visual + auditory) and multi-scale features, enabling the method to maintain high accuracy for both common and rare behaviors in complex scenarios.

[0060] To address the issue that individual tracking in multi-animal scenarios is prone to ID switching or trajectory breaks due to occlusion and cross-movement, thus affecting the continuity of behavior analysis, in S1, the individual animal target detection adopts the YOLOv8n model to obtain a bounding box sequence. ,in For the temporal index of video frames, The total number of video frames. , , , , They represent the first The frame's bounding box's center x-coordinate, center y-coordinate, width, and scheduling; the cross-frame tracking uses the DeepSORT algorithm to output the tracking trajectory. ,in, For individual identification, The time interval for the trajectory.

[0061] The lightweight design of the YOLOv8n model balances speed and accuracy. Through anchoring mechanisms and feature pyramid networks, it can detect multi-scale animal targets in real time, reducing missed detections (especially for small individuals). (Bounding box sequence) It provides accurate spatial positioning, laying the foundation for subsequent feature extraction.

[0062] The DeepSORT algorithm integrates a motion model (Kalman filter predicting location) and an appearance model (feature cosine distance). The Kalman filter uses motion continuity to predict the next frame's location, compensating for short-term occlusion; the Hungarian algorithm associates detection boxes based on Mahalanobis distance and appearance distance, avoiding ID switching. For example, when two animals cross paths, appearance features (such as texture) can distinguish similar individuals, while the motion model handles brief occlusions. (Trajectory) Continuous maintenance ensures the integrity of behavioral data for each individual, enabling subsequent analysis to be based on continuous time-series data and avoiding the loss of behavioral segments due to tracking failures. This directly solves the problem of trajectory fragmentation in multi-animal scenarios and improves the reliability of behavior recognition.

[0063] The above steps, through a specific combination of YOLOv8n and DeepSORT, ensure the stability and accuracy of individual tracking.

[0064] To address the issue that animal behavior may involve both macroscopic movements (such as running) and microscopic actions (such as ear twitching), and that single-scale features cannot fully capture all behavioral patterns, leading to omissions, such as... Figure 3 As shown, in S2, the specific process of multi-scale visual feature extraction includes: extracting the first sub-segment. This is a regular frame rate video clip, the second sub-segment. This is the third sub-segment of the video after frame rate reduction processing. For key parts of the video clips, feature vectors are extracted using three independent ResNet-50 networks. ;in, For video space, , , These are the frame numbers of the extracted first, second, and third sub-segments, respectively. , The height of the video frame. The width of the video frame. The height of a local video segment frame. The width of the local video segment frame, and , , To expand the scaling parameter, the value range can be [0.2, 0.4], where 3 represents the number of channels in the RGB color image. is the dimension of the visual feature vector.

[0065] High frame rates (e.g., 30fps) preserve motion details, and ResNet-50's deep convolutional kernels extract global spatiotemporal features (e.g., motion trajectories), suitable for fast or overall behavior recognition. Lowering the frame rate (e.g., 5fps) filters high-frequency noise and highlights slow or long-term behaviors (e.g., resting postures). ResNet-50 processes data compressed in the time dimension, capturing behavioral trends. Local regions (α-controlled range) focus on key areas; ResNet-50's local receptive fields enhance the extraction of subtle features. For example, At the same time, the mouth area can clearly capture the eating action, while the global view may be obscured.

[0066] Independent ResNet-50 networks avoid cross-scale feature interference, with each network focusing on its own scale characteristics. Output feature vectors. , , By encoding information at different granularities, subsequent fusion ensures that the model responds simultaneously to both macroscopic and microscopic behavioral changes. This addresses the limitations of a single scale, such as in identifying "limp" anomalies, where global fragments... Gait abnormalities were detected, in local segments. Magnify leg movements to collectively improve judgment.

[0067] The above steps achieve multi-level behavioral feature capture by explicitly defining the parameters and network structure of multi-scale sub-segments.

[0068] To address the issue that accurate localization of key areas (such as the mouth and ears) is a prerequisite for local feature extraction, manual or simple bounding box methods are easily affected by pose changes, leading to inaccuracies in local areas. Therefore, the center points of the key areas in the third sub-segment are located using the OpenPose pose estimation algorithm. ,in, , These are the x and y coordinates of the key points in the image, respectively. , For two-dimensional coordinate indexes on a heatmap, For the first Heatmaps of key points, with local areas expanding outwards from these key points at a ratio of [missing information]. .

[0069] OpenPose pose estimation algorithm network structure output heatmap Each pixel value indicates that the location is a key point. The probability. The argmax operation selects the position with the highest probability. This is based on the spatial reasoning capabilities of convolutional networks, which are more robust than traditional methods (such as color segmentation) and can handle occlusion or pose changes. For example, even if the animal's head is tilted, the heatmap can still accurately locate the mouth through skeletal constraints.

[0070] Choosing [0.2, 0.4] balances details and context. If the area is too small, the region will be too narrow, resulting in the loss of associated movements (such as neck movements during chewing). If the value is too large, it will introduce irrelevant background noise. Mathematically, the area of ​​the region is related to... Proportional to this, optimizing computational efficiency.

[0071] Precise positioning ensures local segments By consistently covering behavior-related areas, ResNet-50 extracts more discriminative features. For example, when identifying "head-shaking" anomalies, accurate ear localization can capture high-frequency shaking features, while incorrect localization may include torso interference. This directly improves the quality of local features at the source.

[0072] The above steps use the OpenPose algorithm to mathematically locate key points, ensuring the accuracy of local video segments.

[0073] To address the issue that direct processing of raw audio signals, which contain environmental noise and irrelevant frequency bands, easily leads to feature distortion and affects the extraction of behavior-related sounds, in S3, the Mel spectrogram calculation includes: pre-emphasizing the audio segment ( Frame-by-frame windowing (Hamming window) The amplitude spectrum is obtained by short-time Fourier transform, and then passed through a Mel filter bank. The mapping calculation formula is:

[0074] ;

[0075] in, This is the pre-emphasized audio signal. For discrete audio signals at sampling points The amplitude at that point, For discrete-time indexing, , The length of the audio frame (number of sampling points). For the Hamming window function at point The value at that location, This is the time frame index for the audio spectrogram. Add an index to Mel. For frequency index, For frequency points, For the first in the Mel filter bank Each filter at frequency The weight of the position, For short-time Fourier transform in time and frequency Complex values ​​at that location, For Mel spectrogram in time and Mel bring The energy value at that location, To take the modulus of the complex number.

[0076] High-pass filtering (with a coefficient of up to 0.97) enhances high-frequency components, compensating for sound propagation attenuation and making animal calls (often high-frequency) more prominent. Hamming windows reduce spectral leakage, and STFTs convert the time-domain signal into a time-frequency representation, capturing the temporal changes in sound (such as the duration of calls). Mel filter banks... The triangular weighting simulates the characteristics of human hearing, providing non-uniform focusing for low frequencies (such as breathing sounds) and high frequencies (such as screams). In the calculation formula, the log operation compresses the dynamic range and improves feature stability; the summation operation... Linear frequencies are mapped to a Mel scale, preserving semantic information (such as pitch). The final result is... As image-like features, they are easy for CNNs to process. For example, the pain cries of animals show a concentration of energy in specific frequency bands on the Mel spectrum, while environmental noise is evenly distributed, allowing audio features to effectively distinguish behavioral categories and compensate for the limitations of vision.

[0077] The above steps, through formulaic Mel-frequency spectrum calculation, realize the conversion of sound signals into discriminative features.

[0078] To address the issue that simple splicing or weighted averaging methods cannot capture complex intermodal relationships, potentially leading to information redundancy or conflicts, such as... Figure 4 As shown, in S4, the cross-modal fusion employs a multi-head attention mechanism, specifically including: concatenating multi-scale visual features and then reducing their dimensionality to obtain... Then calculate the output of each attention head:

[0079] ;

[0080] in, The fused visual feature vector To integrate the dimensions of features, For the first The output of each attention head, For the first A query matrix with attention heads For the first The key matrix of each attention head. for The transpose of the matrix, For the first The value matrix of each attention head, For audio feature vectors, For the projection dimension, For the total number of attention heads, For learnable weight matrix, For the first The query projection weight matrix of each attention head For the first The key projection weight matrix of each attention head. For the first The values ​​of each attention head are projected onto the weight matrix. This is the scaling factor;

[0081] The outputs of all heads are concatenated and passed through a linear layer to obtain a fused feature vector.

[0082] visual features As a query, audio features As key / value pairs, through a weight matrix , , Projecting onto a subspace allows different heads to focus on different feature dimensions (e.g., head 1 focuses on time synchronization, head 2 focuses on frequency correlation).

[0083] Calculate the correlation score between visual and auditory features. Scaling prevents gradient vanishing. For example, when visual actions (such as patting) and sounds (such as slapping sounds) occur simultaneously, the weights are automatically increased to strengthen the relevant features.

[0084] Each head independently learns a subspace representation, which is then concatenated to capture multi-dimensional interactions. This is more comprehensive than single-head attention and avoids modality dominance (such as vision overpowering sound).

[0085] Ultimately, the fusion features dynamically integrate bimodal information, reducing conflicts. For example, when recognizing "eating" behavior, visual mouth movements and vocal chewing sounds have greater weight, while wind sounds have less weight. This directly improves the discriminative power of the fused features, supporting more accurate behavior classification.

[0086] The above steps are mathematically implemented through a multi-head attention mechanism to achieve adaptive fusion between modalities and improve feature complementarity.

[0087] To address the problem that traditional classifiers bias the decision boundary towards the majority class when there is class imbalance, leading to misclassification of rare behaviors (such as anomalies), such as... Figure 5 As shown, in S5, the classifier calibration module includes an isoangular tight-frame classifier. and learnable classifiers ,in Satisfy mathematical constraints:

[0088] ;

[0089] in, The weight matrix for the isoangular tight frame classifier is for The transpose of the matrix, For the learnable classifier weight matrix, The total number of behavior categories. for 3D identity matrix for A dimensional vector of all 1s for The transpose of ;

[0090] Final behavior category probability distribution The calculation is as follows:

[0091] , , ;

[0092] in, For learnable equilibrium parameters, The output of the logical value for the isoangular tight frame classifier. The output of the learnable classifier is the logical value. To fuse feature vectors, for The transpose of .

[0093] Isometric tight frame classifier The constraints ensure that the angle between the weight vectors of all classes is maximized (i.e., This geometrically enforces fairness at the decision boundary, ensuring that the classification space is not compressed even if the minority class has fewer samples. For example, At this time, the angle between classes is about 104 degrees, to prevent the majority class from encroaching on the minority class area.

[0094] Learnable classifier Compensation by adapting to the actual distribution through data-driven approaches Theoretical deviations.

[0095] parameter Dynamically balancing the two, in the early stages of training Close to 1, dependency The boundaries of fairness, later Adjustments are made to incorporate data knowledge, enabling the model to maintain high recall even with class imbalance, especially for anomalous behavior. For example, rare "limp" behavior... To obtain a reasonable probability, and By refining the criteria, the imbalance problem was resolved at the decision-making level.

[0096] The above steps optimize the decision boundary through the mathematical constraints and learnable parameters of the isoangular tight frame.

[0097] To address the issue that standard cross-entropy loss, in cases of class imbalance, results in gradient dominance by the majority class, leading to insufficient learning of the minority class, S6 introduces a class-balanced focus loss function. For each sample... The loss is calculated as follows:

[0098] , ;

[0099] in, For training sample index, For the first The true class label of each sample To balance the weights for each category, For the prediction sample Category The probability, This is the aggregation parameter, which can be 2. For categories in the training set The number of samples, This is a hyperparameter and can be set to 0.999.

[0100] in the formula make With sample size Inversely proportional, when When the denominator is small (minority class), Small, The larger the value, the greater the contribution of the minority class loss, forcing the model to focus on the minority class.

[0101] For difficult samples ( (Small) samples are given higher weights, which makes the model focus on misclassified samples and avoids simple samples dominating the gradient.

[0102] Balance the gradients between classes. By balancing intra-class gradients, the loss function prioritizes optimizing minority class and hard samples during training, thereby improving the model's generalization ability. For example, for the abnormal behavior of "screaming," even if the samples are few and prone to misclassification, the loss function ensures that its gradients are effectively backpropagated, thus addressing the imbalance problem at its source.

[0103] The above steps directly adjust the training gradient balance by mathematically transforming the loss function.

[0104] To address the issue that animal behavior exhibits temporal dependence (e.g., the order of walking to running), but the fusion features in S4 lack explicit temporal modeling, potentially leading to a loss of behavioral context, a temporal feature modeling step is added after S4: fusing the feature vectors... Organized into a sequence according to time. Input the bidirectional long short-term memory network and calculate the hidden states:

[0105] ;

[0106] in, For time step index, This represents the total number of time steps. For time step The fused feature vector at that location, For Long Short-Term Memory Networks at time steps The hidden state at that location For Long Short-Term Memory Networks at time steps The hidden state at that location This is the parameter set for Long Short-Term Memory (LSTM) networks. The computational function for the Long Short-Term Memory (LSTM) network, and the hidden state at the last time step. As an enhanced temporal feature vector, it is used for classifier calibration in S5.

[0107] The gating mechanisms of bidirectional long short-term memory networks (input gate, forget gate, output gate) selectively retain long-term dependent memories. The forward long short-term memory network captures behavioral trends from the past to the present (e.g., initiation actions), while the backward long short-term memory network captures context from the future to the present (e.g., stopping intentions). Hidden states are also present. Encoded up to Accumulated information over time.

[0108] The final hidden state Integrate the global temporal context of the entire sequence. For example, when recognizing "chasing" behavior, bidirectional long short-term memory networks can associate features of initial acceleration and continuous running, while single-moment features may be misclassified as isolated actions.

[0109] The above steps enable the model to distinguish similar but temporally different behaviors (such as "walking" and "wandering"), directly solving the problem of insufficient temporal modeling, enhancing the coherence of behavior recognition, and improving the recognition accuracy of continuous behaviors by capturing the dynamic evolution of behavior through a bidirectional long short-term memory network.

[0110] To address the issue that the model decision-making process is often a "black box," making it difficult for users to understand the behavior recognition results and reducing trust and usability, S5, in addition to the existing model, also includes a behavior interpretability analysis step:

[0111] Based on the attention weights of S4 and the classifier weights of S5, the feature contribution is calculated; for the identified abnormal behaviors, the key feature regions and important sound segments are visualized, and text descriptions explaining the causes of the behaviors are generated.

[0112] The attention weights of S4 are used to identify important features within a modality; the classifier weights of S5 are used to identify category-related features; and key regions driving decision-making in the video (such as abnormal leg swings) are located through methods such as gradient class activation maps.

[0113] Key areas are highlighted and associated with sound segments (such as cries of pain) with timestamps. Text descriptions are generated based on feature contribution logic, for example: "Identified as a 'limping' anomaly due to the left leg's range of motion (visual feature) exceeding the normal range, accompanied by groans (audio feature)," enabling users to verify model decisions (e.g., confirming the abnormal location), understand the cause of the behavior (e.g., injury leading to limping), and improve system credibility. Simultaneously, interpretability supports ecologists in in-depth analysis of behavioral patterns, going beyond simple classification functionality.

[0114] The above steps enhance model transparency through interpretability techniques, supporting decision validation and in-depth analysis.

[0115] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An animal behavior recognition method based on multi-scale feature fusion, characterized in that, Includes the following steps: S1. Multimodal Data Synchronous Acquisition and Preprocessing: Simultaneously acquire video stream data and audio stream data containing individual animals through infrared cameras and microphone arrays deployed in the monitoring area; perform animal target detection and cross-frame tracking on the video stream data, and perform noise reduction processing on the audio stream data; S2. Multi-scale visual feature extraction: For each tracked animal individual, extract sub-segments of different scales from its video clips, including complete individual video clips at normal frame rate, video clips processed by reducing frame rate, and local video clips centered on key parts, and extract the corresponding visual feature vectors for each. S3. Audio Feature Extraction and Encoding: Calculate Mel spectrograms for segments of the audio stream data and extract audio feature vectors; S4. Cross-modal feature fusion: After fusing multi-scale visual feature vectors, they are fused with audio feature vectors through an attention mechanism to obtain a fused feature vector; S5. Classifier Calibration and Behavior Recognition: Input the fused feature vector into the calibration module, which contains a fixed-parameter classifier and a learnable-parameter classifier. Obtain the final behavior category probability distribution through the weighted fusion classification result. S6. Model Training and Optimization: Use a class-balanced loss function to train the overall model end-to-end.

2. The animal behavior recognition method based on multi-scale feature fusion according to claim 1, characterized in that, In S1, the individual animal target detection uses the YOLOv8n model to obtain a bounding box sequence. ,in For the temporal index of video frames, The total number of video frames. , , , , They represent the first The frame's bounding box's center x-coordinate, center y-coordinate, width, and scheduling; the cross-frame tracking uses the DeepSORT algorithm to output the tracking trajectory. ,in, For individual identification, The time interval for the trajectory.

3. The animal behavior recognition method based on multi-scale feature fusion according to claim 2, characterized in that, In S2, the specific process of multi-scale visual feature extraction includes: extracting the first sub-segment. This is a regular frame rate video clip, the second sub-segment. This is the third sub-segment of the video after frame rate reduction processing. For key parts of the video clips, feature vectors are extracted using three independent ResNet-50 networks. ;in, For video space, , , These are the frame numbers of the extracted first, second, and third sub-segments, respectively. , The height of the video frame. The width of the video frame. The height of a local video segment frame. The width of the local video segment frame, and , , To expand the scaling parameter, 3 represents the number of channels in the RGB color image. is the dimension of the visual feature vector.

4. The animal behavior recognition method based on multi-scale feature fusion according to claim 3, characterized in that, The center point of the key part of the third sub-segment was located using the OpenPose pose estimation algorithm, and the key point position was determined. ,in, , These are the x and y coordinates of the key points in the image, respectively. , For two-dimensional coordinate indexes on a heatmap, For the first Heatmaps of key points, with local areas expanding outwards from these key points at a ratio of [missing information]. .

5. The animal behavior recognition method based on multi-scale feature fusion according to claim 4, characterized in that, In S3, the Mel spectrogram calculation includes: pre-emphasizing the audio segment, windowing the frame, performing a short-time Fourier transform to obtain the amplitude spectrum, and then passing it through a Mel filter bank. The mapping calculation formula is: ； in, This is the time frame index for the audio spectrogram. Add an index to Mel. For frequency index, For frequency points, For the first in the Mel filter bank Each filter at frequency The weight of the position, For short-time Fourier transform in time and frequency Complex values ​​at that location, For Mel spectrogram in time and Mel bring The energy value at that location, To take the modulus of a complex number.

6. The animal behavior recognition method based on multi-scale feature fusion according to claim 5, characterized in that, In S4, the cross-modal fusion employs a multi-head attention mechanism, specifically including: concatenating multi-scale visual features and then reducing their dimensionality to obtain... Then calculate the output of each attention head: ； in, The fused visual feature vector To integrate the dimensions of features, For the first The output of each attention head, For the first A query matrix with attention heads For the first The key matrix of each attention head. for The transpose of the matrix, For the first The value matrix of each attention head, For audio feature vectors, For the projection dimension, For the total number of attention heads, For learnable weight matrix, For the first The query projection weight matrix of each attention head For the first The key projection weight matrix of each attention head. For the first The values ​​of each attention head are projected onto the weight matrix. This is the scaling factor; The outputs of all heads are concatenated and passed through a linear layer to obtain a fused feature vector.

7. The animal behavior recognition method based on multi-scale feature fusion according to claim 6, characterized in that, In S5, the classifier calibration module includes an isoangular tight-frame classifier. and learnable classifiers ,in Satisfy mathematical constraints: ； in, The weight matrix for the isoangular tight frame classifier is for The transpose of the matrix, For the learnable classifier weight matrix, The total number of behavior categories. for 3D identity matrix for A dimensional vector of all 1s for The transpose of ; Final behavior category probability distribution The calculation is as follows: ， ， ； in, For learnable equilibrium parameters, The output of the logical value for the isoangular tight frame classifier. The output of the learnable classifier is the logical value. To fuse feature vectors, for The transpose of .

8. The animal behavior recognition method based on multi-scale feature fusion according to claim 7, characterized in that, In S6, the loss function is the class-balanced focus loss, for the sample The loss is calculated as follows: ， ； in, For training sample index, For the first The true class label of each sample To balance the weights for each category, For the prediction sample Category The probability, For aggregation parameters, For categories in the training set The number of samples, This is a hyperparameter.

9. The animal behavior recognition method based on multi-scale feature fusion according to claim 8, characterized in that, Following S4, a temporal feature modeling step is also included: fusing the feature vectors. Organized into a sequence according to time. Input the bidirectional long short-term memory network and calculate the hidden states: ； in, For time step index, This represents the total number of time steps. For time steps The fused feature vector at that location, For Long Short-Term Memory Networks at time steps The hidden state at that location For Long Short-Term Memory Networks at time steps The hidden state at that location This is the parameter set for Long Short-Term Memory (LSTM) networks. The computational function for the Long Short-Term Memory (LSTM) network, and the hidden state at the last time step. As an enhanced temporal feature vector, it is used for classifier calibration in S5.

10. The animal behavior recognition method based on multi-scale feature fusion according to claim 9, characterized in that, Following S5, a behavioral interpretability analysis step is also included: The feature contribution is calculated based on the attention weights of S4 and the classifier weights of S5. For the identified abnormal behaviors, key feature regions and important sound segments are visualized, and text descriptions explaining the causes of the behaviors are generated.

Images