Wildlife recognition and behavior analysis monitoring method based on cross-modal data fusion

Abstract

This invention discloses a method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion, belonging to the field of wildlife monitoring technology. The method includes: synchronously acquiring multi-source heterogeneous data with a unified time reference; outputting alignment features through a deformable spatiotemporal alignment module, with the alignment learning rate dynamically adjusted by downstream uncertainty feedback; extracting multi-scale visual and audio features; adding uncertainty prediction branches to each modality, calculating dynamic fusion weights, and enhancing private features; constructing a dynamic spatiotemporal graph, using uncertainty differences as edge weights, and outputting enhanced node features through graph convolution; matching with a multi-prototype comparison library, with temperature parameters dynamically adjusted by uncertainty, outputting behavioral labels or candidate anomalies; and end-to-end optimization of the joint loss function. This invention, with uncertainty as its core, achieves closed-loop collaborative optimization across the entire link from alignment, fusion, graph interaction to identification, significantly improving the robustness and accuracy of wildlife behavior identification in complex field scenarios.

Description

Technical Field

[0001] This invention relates to the field of wildlife monitoring technology, and in particular to a method for wildlife identification and behavioral analysis monitoring based on cross-modal data fusion. Background Technology

[0002] Wildlife behavior monitoring is a key technological support for ecological protection and population management. Currently, data fusion methods based on multimodal sensors such as infrared cameras, visible light cameras, accelerometers, and microphone arrays have become a research hotspot in the field of intelligent wildlife monitoring. However, when existing multimodal sensing technologies are applied to complex field scenarios, they suffer from fundamental defects such as independent functional modules, lack of unified optimization constraints, and isolated propagation and amplification of errors. Furthermore, they lack a reliable measurement and collaborative optimization mechanism that runs through the entire chain of data alignment, feature fusion, interactive modeling, and behavior recognition. A search reveals that while existing publicly available patent technologies have made some progress in multimodal wildlife monitoring, they still have the following specific shortcomings.

[0003] First, spatiotemporal alignment errors are imperceptible and uncorrectable. While Chinese invention patents CN121660505B (Intelligent Monitoring Method for Forest Wildlife Based on Multi-Source Data Fusion) and CN120805034A (A Method for Revealing Animal Behavior in Zoos) employ deformable convolution and skeletal keypoint extraction for cross-modal alignment, their alignment modules are independent of subsequent feature extraction and recognition processes. Alignment errors cannot be perceived or corrected by later modules, resulting in limited alignment accuracy and progressively amplified errors. When interference occurs in the wild, such as rapid animal movement or obstruction by forest branches, causing modal alignment errors to exceed 15°, subsequent fusion and recognition modules cannot perceive this error and continue to perform calculations based on incorrect features, ultimately leading to pose calculation failure and a significant decrease in behavior recognition accuracy.

[0004] Second, the modal fusion weights are statically fixed, and uncertainty information cannot be reused across modules. Although Chinese invention patent CN121660505B proposes a two-branch cross-modal attention network, the fusion weights are fixed after the model training is completed. They cannot be dynamically adjusted according to the real-time quality of the input data (such as sudden changes in illumination and sensor noise). Low-quality modal data will directly pollute the overall fusion features. Although Chinese invention patent CN121302223A (an adaptive multimodal data fusion method for animal scenes) introduces modal confidence calculation based on information entropy, its uncertainty estimation is only used within the fusion module. It does not form a cross-module reverse adjustment closed loop, cannot guide the optimization of preceding alignment, and cannot constrain subsequent interaction modeling. The utilization rate of uncertainty resources is extremely low.

[0005] Third, the mechanisms for handling missing modal data are fragmented and detached from downstream tasks. Existing technologies, such as Chinese invention patents CN121660505B and CN122072704A (an animal behavior recognition method based on multi-scale feature fusion), assume that the monitoring data is complete and usable. They do not systematically model or specifically address issues such as random packet loss and continuous frame loss caused by equipment failure or signal obstruction, and therefore fail directly when data is missing. Although a few solutions use independent pre-completion modules, the completion target only focuses on the low-level accuracy at the pixel or signal level, without linkage and optimization with downstream core tasks such as behavior recognition and group interaction analysis, thus failing to guarantee the effectiveness of actual monitoring tasks.

[0006] Fourth, modal-specific features are easily diluted by common features, and there is no targeted preservation mechanism. Although Chinese invention patent CN122072704A fuses features through multi-scale visual feature extraction and cross-modal attention mechanisms, the fine-grained features unique to each modality (such as audio details and local visual motion) are easily covered and diluted by general common features during the fusion process. Furthermore, there is a lack of targeted preservation and enhancement mechanisms, resulting in insufficient recognition accuracy for fine-grained behaviors such as grooming and chewing in wild animals. Subsequent processing modules are also unable to specifically repair and preserve private features.

[0007] Fifth, individual behavior and group spatial behavior are disconnected, and errors from low-confidence nodes are prone to propagation. Although Chinese invention patent CN121660505B performs spatial overlay analysis in the post-processing stage, it fails to achieve end-to-end joint learning of individual behavioral features and group interaction features. When the monitoring data of a low-confidence individual is noisy, the existing graph interaction model lacks a confidence-based edge weight modulation mechanism, causing the error of that individual to propagate to neighboring individuals through the graph structure, leading to group-level misjudgment propagation.

[0008] In summary, current multimodal wildlife monitoring technologies exhibit systemic drawbacks such as isolated modules, uncontrolled errors, poor adaptability, and feature loss. Each technical module is independently constructed, lacking a unified confidence level as an intermediary for seamless integration and a closed-loop collaborative optimization mechanism. Therefore, there is an urgent need to construct a cross-modal sensing method centered on a unified confidence level and employing end-to-end collaborative optimization to break down the fragmented architecture of each technical module and achieve closed-loop collaborative optimization from data input to behavioral output. Summary of the Invention

[0009] The purpose of this invention is to provide a method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion, which solves the problems of module isolation, error control failure, poor adaptability, feature loss and lack of reliable measurement and closed-loop collaborative optimization mechanism throughout the entire process in existing technologies.

[0010] To achieve the above objectives, this invention provides a method for wildlife identification and behavioral analysis monitoring based on cross-modal data fusion, comprising the following steps: S1. Synchronously collect multi-source heterogeneous data using a unified time benchmark; S2. Input multi-source heterogeneous data into the deformable spatiotemporal alignment module, output aligned multimodal features, and dynamically adjust the learning rate of the deformable spatiotemporal alignment module based on the uncertainty estimate of the output mode in S4. S3. Extract multi-scale visual and audio features from the aligned multimodal features; S4. Configure an uncertainty prediction branch for each modality feature and output the uncertainty estimate of each modality; calculate the dynamic fusion weight based on the uncertainty estimate, obtain the dynamic fusion feature by weighted summation, and then combine the private residual with confidence gating to perform feature compensation to obtain the enhanced fusion feature; S5. Feed back the modal uncertainty estimates output by S4 to S2 to adjust the learning rate of the deformable spatiotemporal alignment module, and pass them to S6. S6. Construct a dynamic spatiotemporal graph, treating each individual wild animal as a graph node, using the enhanced fusion features output from S4 as the initial node features, and employing parameters containing uncertainty difference terms as edge weights; input the constructed dynamic spatiotemporal graph into a spatiotemporal graph convolutional network, and output enhanced node features that fuse individual and group interaction information; S7. Enhance node features and perform distance matching with a multi-prototype comparison library. The temperature parameter of the comparison learning is dynamically adjusted according to the average uncertainty of the node. Finally, wild animal behavior labels or candidate abnormal behaviors are output. S8. Using a joint loss function, perform end-to-end optimization on all learnable parameters from S2 to S7.

[0011] Therefore, the above-mentioned method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion has the following beneficial effects: (1) Construct a full-link closed-loop collaborative optimization mechanism with uncertainty as the core, so that the learning rate of the alignment module can be dynamically adjusted by subsequent fusion uncertainty, the alignment error is not amplified step by step, and the behavior recognition accuracy is significantly improved under interference such as fast movement and occlusion in the wild.

[0012] (2) It realizes the dynamic calculation of modality fusion weights and the targeted retention of private features. Low-quality modalities are automatically downweighted. At the same time, through the confidence-gated private residual addition mechanism, fine-grained features are avoided from being diluted by common features. It has a high accuracy rate in recognizing fine behaviors such as "hair grooming" and "chewing".

[0013] (3) It has robust missing processing capability without the need for an independent completion module. When a certain modality is completely missing, it automatically degenerates into a mode that only relies on reliable modalities. Furthermore, the missing processing is optimized in conjunction with downstream behavior recognition tasks, and the system still works stably under equipment failure or signal obstruction.

[0014] (4) End-to-end joint learning of individual behavior and group spatial behavior. The edge weight introduces the node uncertainty difference term. The error of low confidence node will not spread through the graph structure, effectively suppressing group-level misjudgment, and outputting group indicators such as path efficiency and interaction intensity.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of the wildlife identification and behavior analysis monitoring method based on cross-modal data fusion according to the present invention; Figure 2 The above are comparison diagrams of cross-membrane alignment errors under different alignment methods in the embodiments of the present invention, wherein (a) is a comparison diagram of alignment error changing with time, and (b) is a comparison diagram of alignment error changing with degree of occlusion. Figure 3 This is a heat map of the intensity of wildlife population interaction, as described in an embodiment of the present invention. Detailed Implementation

[0017] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0018] Please see Figures 1-3 A method for wildlife identification and behavioral analysis monitoring based on cross-modal data fusion includes the following steps: S1. Synchronously collect multi-source heterogeneous data with a unified time reference; With a unified time reference as the core, synchronously collect multi-source heterogeneous wildlife monitoring data such as infrared vision, visible light vision, inertial sensing, and audio, ensuring that the timestamps of various modal data are aligned, providing a data foundation for subsequent cross-modal collaborative processing.

[0019] S2. Input multi-source heterogeneous data into the deformable spatiotemporal alignment module, output aligned multimodal features, and dynamically adjust the learning rate of the deformable spatiotemporal alignment module using the uncertainty estimate of the output modes in S4; wherein, the specific method of dynamically adjusting the learning rate of the deformable spatiotemporal alignment module using the uncertainty estimate of the modes output in S4 is as follows: Through feedback function Correcting the offset learning rate of deformable convolution When the uncertainty variance of a certain mode As the size increases, the learning input of the deformable spatiotemporal alignment module for that mode automatically decreases, whereby... This is a correction value for deformable convolution offset; The learning rate is the original offset of the deformable convolution; It is a smooth activation function; For the first Uncertainty variance of each mode; Assign modal numbers; The optimization objective of the deformable spatiotemporal alignment module is to minimize the cross-modal consistency loss. The expression is as follows: ; in, Number the two different modes; It is the L2 norm, used to characterize feature differences; For modality In spatial location ,time Features of the implicit alignment field at the location; Spatial location coordinates; To ensure consistent timestamp collection; the neural implicit alignment field satisfies the following relation: ; in, It is a multilayer perceptron; Spatial location The corresponding spatial implicit encoding; For timestamps The corresponding implicit time encoding.

[0020] S3. Extract multi-scale visual and audio features from the aligned multimodal features; specifically, this includes extracting global visual sub-fragments, down-frame visual sub-fragments, and keypoint-based local visual sub-fragments from the aligned multimodal features, and using a weight-shared ResNet-50 for feature encoding to obtain the visual features. , , ,satisfy And visual feature dimensions ; The synchronously acquired audio data is converted into Mel spectrograms, and then audio features are obtained through encoding using a lightweight convolutional neural network (CNN). ,satisfy And audio feature dimensions ; The aforementioned visual and audio features are input into the spatiotemporal frequency joint attention module, which outputs... The location with the largest value in the spatial attention mask is defined as the spatial anchor point. ; in, , , These are global visual features, frame-reduced visual features, and local visual features, respectively. for 3D real vector space; for 3D real vector space; It is a fine-grained visual-auditory feature set consisting of four modal features; These are the x-coordinate and y-coordinate of the spatial anchor point, respectively.

[0021] S4. Configure an uncertainty prediction branch for each modality feature, and output the uncertainty estimate for each modality; calculate the dynamic fusion weights based on the uncertainty estimates, obtain the dynamic fusion features by weighted summation, and then perform feature compensation by combining the private residuals of confidence gating to obtain the enhanced fusion features; the specific calculation method of the uncertainty prediction branch is as follows: two fully connected layers are connected in series at the output of the feature encoder of each modality. The first fully connected layer compresses the feature dimension to 512, and the second fully connected layer outputs a scalar. The modal uncertainty variance is obtained through exponential mapping operations, and the calculation formula is as follows: ; in, This represents the logarithmic variance of the network output. It is an exponential function; For the first Uncertainty variance of each mode.

[0022] S4 specifically includes: S41, Modal Number These correspond to the global visual modality, the reduced-frame visual modality, the local visual modality, and the audio modality, respectively; each modality feature... Configure an uncertainty prediction branch to output the uncertainty variance of the corresponding mode. ; S42. Calculate the dynamic fusion weights for each modality, expressed as: ; in, For the first Dynamic fusion weights for each modality; For learnable inverse temperature parameters; For modal traversal indices; S43. Calculate the dynamic fusion feature based on the dynamic fusion weight, the expression is: ; in, The dynamic fusion feature is obtained by multimodal weighted summation; S44. Enhanced fusion features are obtained by superimposing modal private residuals through confidence gating, as shown in the formula: ; in, To ultimately enhance fusion features; It is the hyperbolic tangent activation function; This is a linear mapping layer, whose function is to project the fused features onto the first... The feature space of each modality; For the first Private residual features for each modality.

[0023] Feature fusion also includes adaptive modality data missing fault tolerance processing, specifically: when a certain modality data is completely missing, the modality features are... Set to the zero vector of the same dimension, the modal uncertainty is satisfied by the learnable bias constraint. This makes the modal fusion weights The overall framework adaptively switches to a working mode that uses only valid and reliable modalities; in the case of missing data, it enhances the fusion features. The calculation formula is: ; in, The traversal index for the effective modal; Representing positive infinity, it indicates that the modal data is completely unreliable; this method does not require additional configuration of a separate data missing completion module and is used to adapt to abnormal working conditions in which single or multiple modal data are completely missing during field monitoring.

[0024] S5. Feedback the modal uncertainty estimates output from S4 to S2 to adjust the learning rate of the deformable spatiotemporal alignment module, and simultaneously pass them to S6; specifically: feed back the modal uncertainty estimates output from S4 to S2 to adjust the learning rate of the deformable spatiotemporal alignment module. The reverse transmission is divided into two paths: one path feeds back to the deformable spatiotemporal alignment module of S2, dynamically corrects the alignment learning rate, and realizes closed-loop optimization of "fusion quality constraint alignment accuracy"; the other path directly transmits to the subsequent spatiotemporal graph modeling module, providing confidence basis for adaptive modulation of group interaction weights and realizing collaborative constraint of confidence across all modules.

[0025] S6. Construct a dynamic spatiotemporal graph, treating each individual wild animal as a graph node, using the enhanced fusion features output from S4 as the initial node features, and employing parameters containing uncertainty difference terms as edge weights; input the constructed dynamic spatiotemporal graph into a spatiotemporal graph convolutional network, outputting enhanced node features that fuse individual and group interaction information; the edge weight calculation formula for the dynamic spatiotemporal graph is as follows: ; in, For graph nodes With nodes Edge weights between them; These are two nodes representing individual wild animals in a dynamic spatiotemporal graph; For nodes The spatial Euclidean distance of the corresponding individual; The spatial distance is represented by the Gaussian kernel variance. For nodes The angle between the eigenvectors; The cosine value of the characteristic included angle; The occlusion coefficient has a value range of [0,1]. This represents the absolute value of the difference between the average uncertainties of the two nodes. For reference, the variance of uncertainty; The feature enhancement factor is determined by the spatial anchor point; node The formula for calculating the average uncertainty is: ; in, For nodes Corresponding to the Uncertainty variance of each mode; The single-layer update formula for a spatiotemporal graph convolutional network is: ; in, For the first Nodes output by layer graph convolution feature; For the first Nodes updated after layer graph convolution feature; It is an exponential linear activation function; For the first Learnable weight matrix for layer graph convolution; For nodes The set of neighboring nodes; They are nodes ,node The number of neighboring nodes; The graph comparison loss is: ; in, To compare the losses; It is the set of all edges in the dynamic spatiotemporal graph; This represents the total number of edges in the graph; These are the nodes output by the second layer graph convolution. Features; For feature similarity function; To compare and contrast the temperature coefficients; The index of the negative sample node.

[0026] S7. Enhance node features are matched with a multi-prototype comparison library to determine distance. The temperature parameter for comparison learning is dynamically adjusted based on the average uncertainty of the nodes. The final output is a wildlife behavior label or candidate abnormal behavior. Specifically, wildlife behavior classification and abnormal behavior determination are achieved through multi-prototype comparison learning. The process involves setting the total number of wildlife behavior categories. Learning for each type of behavior prototype vectors ,in Number the behavior category. For prototype numbers under a single category; Targeting node features The formula for class-balanced contrastive loss is: ; in, For a single node Prototype contrast loss; For nodes eigenvectors; For nodes The corresponding real-world behavior category; For nodes The nearest neighbor prototype vector that was matched; Traverse the index for behavior categories; For nodes The corresponding dynamic temperature parameters; For nodes Category balance weight coefficients; The formula for calculating dynamic temperature parameters is: ; The formula for calculating the average uncertainty at each node is: ; The formula for calculating the category balance weight coefficient is: ; in, The base temperature coefficient; This is the uncertainty adjustment coefficient; For nodes The average uncertainty; For nodes Corresponding to the Uncertainty variance of each mode; This is the variance calculation function; The characteristic variance normalization coefficient; Represents a node All domain nodes The set of feature vectors output after the second layer of graph convolution; The rule for determining abnormal behavior is: when This is identified as a candidate abnormal behavior; among them, This means iterating through all behavior categories and their prototypes and taking the maximum similarity. The threshold for determining abnormal behavior.

[0027] S8. Using a joint loss function, perform end-to-end optimization on all learnable parameters from S2 to S7; where the joint loss function is: ; in, Joint loss optimized for end-to-end; This represents the overall prototype comparison loss; For cross-modal consistency loss; For modal uncertainty loss; To compare the losses; This is the weighted regularization loss; The balancing coefficients for the various losses are, in order of importance. , , , ; The formula for calculating modal uncertainty loss is: ; in, The mean vector of the four modal features is calculated using the following formula: ; The formula for calculating weighted regularization loss is: ; in, For all learnable weight matrices of the network The square of the L2 norm; The formula for calculating the overall prototype contrast loss is: ; in, It is the set of all nodes in the dynamic spatiotemporal graph; The total number of nodes; This method uses the Adam optimizer to complete end-to-end training of the entire network. The learnable parameters include all parameters from S2 to S7 that need to be automatically updated through network training rather than manually set.

[0028] S9. Combining the dynamic spatiotemporal graph structure of S6 with the behavioral labels output from S7, calculate the spatial behavior indicators of wild animal groups. The specific indicators are as follows: (1) Path efficiency index, expressed as: ; in, The path efficiency index; For a single wild animal in a time window The actual path length within minutes; The straight-line distance between the starting and ending points of a wild animal's movement; (2) The intensity of interaction between individuals is expressed as: ; in, For nodes The intensity of interaction between the two wild animals; To satisfy edge weight And the number of events with the behavioral tag being a social category; The duration of continuous interaction between the two animals; Length of time for the president; For time decay weight, , This is the time offset of the event. This is the time decay coefficient; (3) The nuclear density estimation heatmap is expressed as: ; in, Spatial location The population nuclear density value at that location; This represents the total number of animal location samples. For the first The spatial coordinates of each animal; Meters represent the kernel density bandwidth; The square of the L2 distance between the coordinate points; By overlaying individual wildlife behavior tags with the aforementioned spatial behavior indicators, a comprehensive monitoring report is ultimately generated.

[0029] Example I. Implementation Scenarios and Experimental Configuration This embodiment selects a wild macaque population in a forest area as the monitoring target. The monitoring area is a natural broad-leaved forest in southern China, where the environment presents typical complex interference scenarios such as foliage obstruction, changes in light intensity, rapid animal movement, and mixed audio sounds like wind and insect chirping. A multimodal acquisition device consisting of visible light cameras, infrared cameras, and a microphone array is deployed to... To unify the time benchmark, visual and audio heterogeneous data from multiple sources were collected synchronously, with a total monitoring time of 10 hours. The focus was on identifying, analyzing, and monitoring the group status of macaques, particularly their fine motor behaviors such as grooming and chewing, as well as routine behaviors such as walking, resting, chasing, and social interaction.

[0030] The model is built according to the patented technology solution and includes a deformable spatiotemporal alignment module, a multi-scale feature extraction module, a modal uncertainty prediction branch, a dynamic feature fusion module, a dynamic spatiotemporal graph convolutional network, and a multi-prototype contrastive classification library; the total number of preset behavior categories is also included. Number of prototypes in a single category Abnormal behavior judgment threshold Set to 0.3, the loss balance coefficient for each item. .

[0031] II. Implementation of the complete process Step 1: Synchronous acquisition of multi-source data: Based on a unified timestamp, four types of modal data of macaques in the forest area are collected simultaneously: global vision, frame-reduced vision, local key point vision, and audio. Visual data captures the macaques' limb movements, facial and fur details, while audio data collects characteristic signals such as chewing sounds, limb rubbing sounds while grooming, and calls, thus completing the collection of multi-source heterogeneous data.

[0032] Step 2, Cross-modal spatiotemporal alignment and dynamic learning rate adjustment: Input the four types of modal data into the deformable spatiotemporal alignment module to complete the spatiotemporal feature alignment. At the same time, the modal uncertainty estimate output by the subsequent module is fed back to this module to dynamically correct the deformable convolution offset learning rate: When the visual modality experiences a decrease in data quality and an increase in uncertainty variance due to occlusion of branches and leaves, the module automatically reduces the learning input for that modality; the audio modality signal is stable and has low uncertainty, so it maintains normal learning intensity.

[0033] like Figure 2 , Figure 2 (a) in the figure shows the curve of alignment error changing over time. Figure 2 (b) shows the alignment error as a function of occlusion, indicating that the alignment scheme based on the uncertainty-dynamically adjusted learning rate of this invention, compared with the traditional fixed learning rate method, maintains a low cross-modal alignment error under conditions of long-term monitoring and increased occlusion, effectively solving the problem of gradual amplification of alignment error in traditional techniques, and laying the foundation for subsequent fine behavioral feature extraction.

[0034] Step 3: Multi-scale visual and audio feature extraction: Visual dimension: Global visual sub-fragments, frame-reduced visual sub-fragments, and local visual sub-fragments based on body / facial key points of macaques were extracted respectively. Through weight-sharing ResNet-50 encoding, three sets of 2048-dimensional visual features were obtained. Among them, the local visual features mainly retain fine-grained motion information such as hair movement and mouth chewing. Audio Dimensions: The original audio is converted into a Mel spectrogram, and then encoded with a lightweight CNN to obtain 512-dimensional audio features, accurately preserving weak audio signals such as chewing friction sounds and grooming limb touch sounds; All features are input into the spatiotemporal frequency joint attention module, which outputs four sets of fine-grained visual-auditory feature sets and locks the region with the strongest spatial attention as the spatial anchor point.

[0035] Step 4: Uncertainty prediction and dynamic feature fusion, private feature enhancement: Uncertainty prediction branches are configured for the four sets of modal features, and the uncertainty variance of each modality is calculated. Dynamic fusion weights are calculated based on the variances, with the weights of low-quality modalities being automatically reduced and the weights of high-confidence modalities being increased. After completing the multimodal feature weighted fusion, the unique fine-grained features of each modality are superimposed into the fused features through confidence gating and private residual compensation mechanism: the visual modality retains the local details of macaque grooming and mouth opening and closing, and the audio modality retains the exclusive voiceprint features of chewing and grooming, so as to avoid the dilution of fine-grained features by general common features. The simulation equipment was designed to handle short-term failure conditions. A single visual modality was artificially set to be missing, and the uncertainty of this modality approached infinity. The fusion weight was reduced to 0, and the model adaptively switched to work with the remaining effective modalities, without any recognition interruption.

[0036] Step 5: Dynamic Spatiotemporal Graph Construction and Group Interaction Feature Modeling: Using each macaque in the forest area as a graph node, the enhanced fusion features output in step 4 are used as the initial features of the node. When constructing the dynamic spatiotemporal graph, the edge weights incorporate parameters such as node uncertainty differences, spatial distance, occlusion coefficient, and feature angle. For individuals with high uncertainty (poor data quality), the edge weights between them and surrounding nodes are reduced to prevent the spread of errors to the group.

[0037] The spatiotemporal graph is input into a graph convolutional network, which integrates information on individual macaque behavior and group interaction, and outputs enhanced node features. This not only preserves the individual behavioral features of individual macaques grooming and chewing, but also captures the group social behavior of macaques grooming each other.

[0038] Step 6: Multi-prototype comparison and classification, and abnormal behavior determination: The enhanced node features are compared with a similarity database consisting of 8 types of behaviors, each with 5 sets of prototype vectors. The temperature parameter of the comparison learning is dynamically adjusted according to the average uncertainty of the nodes. Fine-grained behavior recognition performance: For grooming behavior, the model relies on local visual hair movement features and limb friction audio features, achieving a similarity of over 0.87 with the corresponding prototype vector; for chewing behavior, it relies on visual features of key points in the mouth and chewing voiceprint features, achieving a similarity of over 0.89; the accuracy rates for the two types of fine-grained behavior recognition reached 96.2% and 95.7% respectively, far exceeding the traditional static fusion model (the accuracy rate of the traditional model for both types of behavior recognition is less than 80%).

[0039] When the feature has a maximum similarity to all prototypes that is below a threshold When such behavior is deemed abnormal (e.g., a macaque monkey exhibiting unusual behavior when injured, or a strange creature intruding), it is considered abnormal behavior.

[0040] Step 7, End-to-end joint optimization: Using joint loss function It integrates cross-modal alignment loss, modal uncertainty loss, graph contrast loss, prototype contrast loss and weight regularization loss, and uses the Adam optimizer to perform end-to-end iterative optimization of all network parameters, further enhancing the end-to-end collaborative capability.

[0041] Step 8: Calculation and visualization of group spatial behavior indicators: Combining the dynamic spatiotemporal graph structure and individual behavioral labels, indicators for three groups were calculated: Path efficiency index: Analyzing the movement path patterns of macaques; Intensity of individual interaction: The frequency and duration of social behaviors such as grooming among macaques were statistically analyzed; Population nuclear density estimation: generating spatial distribution heatmaps.

[0042] like Figure 3 As shown, this figure is a heatmap of macaque population interaction intensity. The legend distinguishes between low and high interaction levels, and the figure marks the distribution areas of different macaque populations A, B, C, W, X, Y, and Z. Combining color and gradient labels, it is evident that the interaction intensity is higher in areas where populations W, Y, and B are located, while the interaction intensity is relatively lower in areas where populations A, C, Z, and X are located. There are significant differences in the degree of macaque aggregation within these corresponding areas, and macaques exhibit more frequent fine social behaviors such as grooming in high-interaction areas. This invention can accurately delineate the active interaction areas of different populations, clearly distinguish the differences in group behavior in each area, and successfully achieve linked analysis of individual behavior and population status.

[0043] Therefore, this invention adopts the aforementioned wildlife identification and behavior analysis monitoring method based on cross-modal data fusion, with uncertainty estimation as the core, running through the entire chain of data alignment, feature fusion, graph interaction, and behavior recognition. First, multi-source heterogeneous data is synchronously collected using a unified time base and input into a deformable spatiotemporal alignment module. The learning rate of this module is dynamically adjusted by the modal uncertainty of the downstream output, achieving closed-loop correction of alignment errors. Next, multi-scale visual and audio features are extracted, and an uncertainty prediction branch is added to each modality, outputting the uncertainty estimate for each modality. Based on this, dynamic fusion weights are calculated. Simultaneously, a confidence-gated private residual addition mechanism enhances the modal private features back into the fusion features, effectively preserving fine-grained information. Then, a dynamic spatiotemporal graph is constructed, with individuals as nodes and enhanced fusion features as initial features. Edge weights include uncertainty difference terms, and individual-group interaction features are output through spatiotemporal graph convolution. Finally, distance matching is performed with a multi-prototype comparison library, and the temperature parameter is dynamically adjusted by the average uncertainty of the nodes, outputting behavioral labels or candidate abnormal behaviors. By optimizing all learnable parameters end-to-end using a joint loss function, low-quality modes are automatically downweighted and the propagation of highly uncertain individual graphs is restricted, significantly improving the robustness of recognition in complex field scenarios.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for wildlife identification and behavioral analysis monitoring based on cross-modal data fusion, characterized in that, Includes the following steps: S1. Synchronously collect multi-source heterogeneous data using a unified time benchmark; S2. Input multi-source heterogeneous data into the deformable spatiotemporal alignment module and output aligned multimodal features; S3. Extract multi-scale visual and audio features from the aligned multimodal features; S4. Configure an uncertainty prediction branch for each modality feature and output the uncertainty estimate of each modality; calculate the dynamic fusion weight based on the uncertainty estimate, obtain the dynamic fusion feature by weighted summation, and then combine the private residual with confidence gating to perform feature compensation to obtain the enhanced fusion feature; S5. Feed back the modal uncertainty estimates output by S4 to S2 to adjust the learning rate of the deformable spatiotemporal alignment module, and pass them to S6. S6. Construct a dynamic spatiotemporal graph, taking each individual wild animal as a graph node, using the enhanced fusion features output by S4 as the initial features of the nodes, and using parameters containing uncertainty difference terms as edge weights. The constructed dynamic spatiotemporal graph is input into the spatiotemporal graph convolutional network, and the output is an enhanced node feature that integrates individual and group interaction information; S7. Enhance node features and perform distance matching with a multi-prototype comparison library. The temperature parameter of the comparison learning is dynamically adjusted according to the average uncertainty of the node. Finally, wild animal behavior labels or candidate abnormal behaviors are output. S8. Using a joint loss function, perform end-to-end optimization on all learnable parameters from S2 to S7.

2. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 1, characterized in that, In S2, the specific method for dynamically adjusting the learning rate of the deformable spatiotemporal alignment module using the modal uncertainty estimate output by S4 is as follows: Through feedback function Correcting the offset learning rate of deformable convolution When the uncertainty variance of a certain mode As the size increases, the learning input of the deformable spatiotemporal alignment module for that mode automatically decreases, whereby... This is a correction value for deformable convolution offset; The learning rate is the original offset of the deformable convolution; It is a smooth activation function; For the first Uncertainty variance of each mode; Assign modal numbers; The optimization objective of the deformable spatiotemporal alignment module is to minimize the cross-modal consistency loss. The expression is as follows: ； in, Number the two different modes; It is the L2 norm, used to characterize feature differences; For modality In spatial location ,time Features of the implicit alignment field at the location; Spatial location coordinates; To ensure consistent timestamp collection; the neural implicit alignment field satisfies the following relation: ； in, It is a multilayer perceptron; Spatial location The corresponding spatial implicit encoding; For timestamps The corresponding implicit time encoding.

3. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 2, characterized in that, S3 specifically includes: extracting global visual sub-fragments, down-frame visual sub-fragments, and keypoint-based local visual sub-fragments from the aligned multimodal features, and using a weight-shared ResNet-50 for feature encoding to obtain visual features. , , ,satisfy And visual feature dimensions ; The synchronously acquired audio data is converted into Mel spectrograms, and then audio features are obtained through encoding using a lightweight convolutional neural network (CNN). ,satisfy And audio feature dimensions ; The aforementioned visual and audio features are input into the spatiotemporal frequency joint attention module, which outputs a detailed... The location with the largest value in the spatial attention mask is defined as the spatial anchor point. ; in, , , These are global visual features, frame-reduced visual features, and local visual features, respectively. for 3D real vector space; for 3D real vector space; It is a fine-grained visual-auditory feature set consisting of four modal features; These are the x-coordinate and y-coordinate of the spatial anchor point, respectively.

4. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 3, characterized in that, The specific calculation method of the uncertainty prediction branch in S4 is as follows: two fully connected layers are connected in series at the output of the feature encoder of each modality. The first fully connected layer compresses the feature dimension to 512, and the second fully connected layer outputs a scalar. The modal uncertainty variance is obtained through exponential mapping operations, and the calculation formula is as follows: ； in, This represents the logarithmic variance of the network output. It is an exponential function; For the first Uncertainty variance of each mode.

5. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 4, characterized in that, S4 specifically includes: S41, Modal Number These correspond to the global visual modality, the reduced-frame visual modality, the local visual modality, and the audio modality, respectively; each modality feature... Configure an uncertainty prediction branch to output the uncertainty variance of the corresponding mode. ; S42. Calculate the dynamic fusion weights for each modality, expressed as: ； in, For the first Dynamic fusion weights for each modality; For learnable inverse temperature parameters; For modal traversal indices; S43. Calculate the dynamic fusion feature based on the dynamic fusion weight, the expression is: ； in, The dynamic fusion feature is obtained by multimodal weighted summation; S44. Enhanced fusion features are obtained by superimposing modal private residuals through confidence gating, as shown in the formula: ； in, To ultimately enhance fusion features; It is the hyperbolic tangent activation function; This is a linear mapping layer, whose function is to project the fused features onto the first... The feature space of each modality; For the first Private residual features for each modality.

6. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 5, characterized in that, S4's feature fusion also includes adaptive modality data missing fault tolerance processing, specifically: when a certain modality's data is completely missing, the modality's features are... Set to the zero vector of the same dimension, the modal uncertainty is satisfied by the learnable bias constraint. This makes the modal fusion weights The overall framework adaptively switches to a working mode that uses only effective and reliable modes; Enhanced fusion features in the case of missing data The calculation formula is: ； in, The traversal index for the effective modal; Representing positive infinity, it indicates that modal data is completely unreliable.

7. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 6, characterized in that, The formula for calculating the edge weights of the dynamic spatiotemporal graph in S6 is as follows: ； in, For graph nodes With nodes Edge weights between them; These are two nodes representing individual wild animals in a dynamic spatiotemporal graph; For nodes The spatial Euclidean distance of the corresponding individual; The spatial distance is represented by the Gaussian kernel variance. For nodes The angle between the eigenvectors; The cosine value of the characteristic included angle; The occlusion coefficient has a value range of [0,1]. This represents the absolute value of the difference between the average uncertainties of the two nodes. For reference, the variance of uncertainty; The feature enhancement factor is determined by the spatial anchor point; node The formula for calculating the average uncertainty is: ； in, For nodes Corresponding to the Uncertainty variance of each mode; The single-layer update formula for a spatiotemporal graph convolutional network is: ； in, For the first Nodes output by layer graph convolution feature; For the first Nodes updated after layer graph convolution feature; It is an exponential linear activation function; For the first Learnable weight matrix for layer graph convolution; For nodes The set of neighboring nodes; They are nodes ,node The number of neighboring nodes; The graph comparison loss is: ； in, To compare the losses; It is the set of all edges in the dynamic spatiotemporal graph; This represents the total number of edges in the graph; These are the nodes output by the second layer graph convolution. Features; For feature similarity function; To compare and contrast the temperature coefficients; The index of the negative sample node.

8. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 7, characterized in that, In S7, wildlife behavior classification and abnormal behavior identification are achieved through multi-prototype contrastive learning. The specific process is as follows: Set the total number of wildlife behavior categories. Learning for each type of behavior prototype vectors ,in Number the behavior category. For prototype numbers under a single category; Targeting node features The formula for class-balanced contrastive loss is: ； in, For a single node Prototype contrast loss; For nodes eigenvectors; For nodes The corresponding real-world behavior category; For nodes The nearest neighbor prototype vector that was matched; Traverse the index for behavior categories; For nodes The corresponding dynamic temperature parameters; For nodes Category balance weight coefficients; The rule for determining abnormal behavior is: when This is identified as a candidate abnormal behavior; among them, This means iterating through all behavior categories and their prototypes and taking the maximum similarity. The threshold for determining abnormal behavior.

9. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 8, characterized in that, The joint loss function in S8 is: ； in, Joint loss optimized for end-to-end; This represents the overall prototype comparison loss; For cross-modal consistency loss; For modal uncertainty loss; To compare the losses; This is the weighted regularization loss; This is the balance coefficient for all losses.

10. The method for wildlife identification and behavior analysis monitoring based on cross-modal data fusion according to claim 9, characterized in that, It also includes S9: Combining the dynamic spatiotemporal graph structure of S6 with the behavioral labels output from S7, spatial behavior indicators of wildlife groups are calculated, and the specific indicators are as follows: (1) Path efficiency index, expressed as: ； in, The path efficiency index; This represents the actual movement path length of a single wild animal within the time window. The straight-line distance between the starting and ending points of a wild animal's movement; (2) The intensity of interaction between individuals is expressed as: ； in, For nodes The intensity of interaction between the two wild animals; To satisfy edge weight And the number of events with the behavioral tag being a social category; The duration of continuous interaction between the two animals; Length of time for the president; For time decay weight, , This is the time offset of the event. This is the time decay coefficient; (3) The nuclear density estimation heatmap is expressed as: ； in, Spatial location The population nuclear density value at that location; This represents the total number of animal location samples. For the first The spatial coordinates of each animal; For kernel density bandwidth; The L2 distance between the coordinate points is squared.

Images