Multi-mode wild animal action intelligent retrieval and positioning method

Abstract

The invention discloses a multi-modal wild animal action intelligent retrieval and positioning method, and aims to realize active discovery and accurate positioning of behaviors in an open scene. The method comprises the following steps: firstly, extracting spatio-temporal features from a monitoring video through an action recognition module in combination with a TimeSform model, and generating an interactive action category library; then, vision-language mapping is carried out by utilizing a CLIP model, an'action-text 'candidate description pair is generated, and a structured behavior library is constructed through K-means clustering; and then, based on semantic description or natural language query selected by a user from the behavior library, positioning a corresponding space-time fragment in the video by adopting a bidirectional semantic alignment module and a space-time attention mechanism, and outputting a behavior category, a time interval and a space bounding box. And finally, generating a structured result through joint optimization of an action recognition module and a video grounding module. The method breaks through the limitation of traditional'passive response known query ', realizes a new normal form of'active perception-accurate query', remarkably improves the behavior discovery rate and the positioning accuracy, and is suitable for the fields of ecological protection, intelligent security and protection and the like.

Description

Technical Field

[0001] This invention relates to the field of intelligent wildlife monitoring and multimodal video understanding, specifically an intelligent method for retrieving and locating wildlife behavior videos, used to proactively discover and accurately locate target behaviors in massive amounts of video data. Background Technology

[0002] Wildlife behavior research is a core component of ecological conservation and biodiversity maintenance. Analyzing animal behavior patterns provides a scientific basis for developing conservation strategies. Traditional monitoring methods rely on manual inspections and video annotation, which face three major bottlenecks: First, the massive amount of video data far exceeds human processing capacity, leading to a high rate of missed detections of key behaviors; second, complex environments increase the subjectivity of human judgment, resulting in insufficient accuracy in behavior classification; and finally, passive response models cannot meet the needs of real-time monitoring, such as poaching, which is often discovered only after the fact, missing the opportunity for intervention.

[0003] In recent years, video grounding technology has significantly improved the efficiency of behavior localization by locating spatiotemporal segments in videos based on natural language queries through visual-linguistic cross-modal mapping, reducing the time taken from hours to seconds, and improving localization accuracy in controlled scenarios. However, existing technologies assume that users can provide precise behavioral descriptions, while in real-world scenarios, users often need to first discover unknown behaviors and then query their activity trajectories. This "targetless query" requirement and the challenge of visual-linguistic modal alignment in complex environments make traditional video grounding technology difficult to leverage in open scenarios.

[0004] To address the shortcomings of traditional methods in terms of efficiency, accuracy, and real-time performance, as well as the core limitations of existing video grounding technologies in supporting "targetless queries" and modal alignment in complex environments, this study proposes an intelligent solution that integrates action recognition and video grounding technologies. This solution constructs a "discover first, query later" multimodal video understanding framework: the action recognition module proactively perceives unknown behaviors in the video and generates an interactive action category library, overcoming the high false negative rate and strong subjectivity of traditional methods; the video grounding module enables accurate mapping between natural language queries and video spatiotemporal segments; and the joint optimization of the two modules addresses the cross-modal alignment challenges in complex environments. Ultimately, this innovative model of "proactively discovering unknown behaviors first, then accurately locating activity trajectories" not only meets the needs for automated and efficient processing of massive amounts of data but also enables real-time monitoring and proactive intervention, fully adapting to the practical needs of open-scene wildlife monitoring. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a deep learning-based method for retrieving and locating wildlife behavior videos. It aims to achieve a new paradigm of video understanding—"active perception → precise query"—through the deep integration of action recognition and video grounding technologies, thereby solving the technological gap in behavior discovery and localization in open scenarios.

[0006] To achieve the above objectives, this invention provides a method for retrieving and locating wildlife behavior videos based on deep learning, characterized by the following steps:

[0007] Step 1: Action Recognition and Behavior Database Generation

[0008] The action recognition module actively senses various behaviors in the video and generates an interactive action category library, providing a semantic foundation for subsequent queries.

[0009] A0. Using a finely tuned motion recognition model on a wildlife video dataset, extract visual features from the input video stream to identify potential behavior categories;

[0010] A1. Based on the vision-language pre-trained model, visual features are mapped to the language semantic space to generate "action-text" candidate description pairs;

[0011] A2. The behavior categories are normalized using a clustering algorithm to generate a structured action category library for interactive user queries.

[0012] Step 2: Semantic query and video grounding.

[0013] Based on the semantic description selected by the user from the action category library, the video grounding module is triggered to locate the corresponding spatiotemporal segment in the video.

[0014] B0. Receive natural language queries input by the user, or select a target behavior description from the action category library;

[0015] B1. Utilize a bidirectional semantic alignment module to perform cross-modal matching between language queries and video visual features, thereby enhancing the association weight between action-related words and visual features.

[0016] B2. Using a spatiotemporal attention mechanism, locate and query matching spatiotemporal segments in the video stream, and output the specific time interval and spatial region where the behavior occurred.

[0017] Step 3: End-to-end optimization and output.

[0018] By jointly optimizing the action recognition and video grounding modules, the synergy between behavior detection and localization is ensured.

[0019] C0. An end-to-end training strategy is adopted to optimize the behavior detection rate of the action recognition module and the positioning accuracy of the video grounding module;

[0020] C1. Generate structured output containing behavior categories, time intervals, and spatial regions for user analysis or subsequent decision-making.

[0021] Furthermore, step one, A0 specifically includes:

[0022] A0, For the input video V∈R T×H×W×3 Perform frame sampling and extract the RGB frame sequence. and optical flow characteristics

[0023] A1. Using a pre-trained TimeSformer model, fine-tuned with a wildlife dataset, to extract spatiotemporal features (F) from video frames. v =TimeSformer(V; θ) v )∈R T×512 ), where (θ v ) is the feature dimension.

[0024] A2. Predict the behavior category using the classification head, and output the probability distribution P(c|V) = Softmax(W). c F v +b c ), where W c ∈R C×D b c ∈R C Here are the classification layer parameters, and C is the number of behavior categories. Behavior categories with a confidence level higher than 0.7 are selected.

[0025] C selected =c i |(c i |)>0.7

[0026] Furthermore, step A1 in step one specifically includes:

[0027] B0. Extracting high-level visual features from video frames using a CLIP model visual encoder. The formula is:

[0028] F clip =CLIP visual (V;θ clip B1. Visual features F clip The input CLIP text encoder generates a predefined behavior description template T = {t1, t2, ..., t...} M The matched candidate text. Text features are:

[0029]

[0030] B2. Calculate the semantic alignment between visual features and text features using cosine similarity:

[0031]

[0032] Retain action-text pairs with a similarity greater than 0.8:

[0033] Pairs = {(c i ,t j )|S(F clip ,F t >0.8

[0034] Furthermore, step A2 in step one specifically includes:

[0035] C0, Features of the generated "action-text" pairs {F t K-means clustering was performed, and redundant descriptions were merged based on semantic similarity. The cluster centers were determined to be... Where k = 1, ..., K is the number of clusters;

[0036] C1. Construct an action category library L = {(c i ,F t ,μ k )}, storing behavior categories, corresponding text descriptions, and feature vectors;

[0037] C2. Provides an interactive interface that presents an action category library through keyword search, allowing users to browse and select.

[0038] Furthermore, step B1 in step two specifically includes:

[0039] D0. Encode the user-input language query Q using the BERT model and extract its textual semantic features. The formula is:

[0040] F q =BERT(Q; θ) bert )

[0041] D1. Design a bidirectional semantic alignment module to enhance the association between action-related words and video visual features F_v in language queries through a multi-head attention mechanism. The attention calculation formula is:

[0042]

[0043] Where, query Q=W q F q Key K = W k F v Value V = W v F v W q Wk W v ∈R D×D d is a learnable projection matrix. k The dimension of the key;

[0044] D2. Generate cross-modal feature representations F cross ∈R T×D It is used for subsequent spatiotemporal positioning.

[0045] Furthermore, step B2 in step two specifically includes:

[0046] E0, Spatiotemporal features based on video frames F v Construct a 3D feature map M∈R T×H′×W′ Where H′, W′ are the spatial downsampling resolutions;

[0047] E1. Utilizing a spatiotemporal attention mechanism, combined with cross-modal features F cross Calculate the matching score for each spatiotemporal region:

[0048] S st =Softmax(W s (M·F cross )+b s ),W s ∈R D×1 ,b s ∈R

[0049] E2. Output the spatiotemporal segment with the highest matching score, including the time interval [t]. s ,t e The bounding box B = (x,y,w,h) is defined by the bounding box, where (x,y) are the normalized center coordinates and (w,h) are the bounding box dimensions.

[0050] Furthermore, step three, specifically C0, includes:

[0051] F0. Design a joint loss function that fuses the classification loss L from action recognition. cls Positioning loss L with video grounding loc The classification loss uses cross-entropy:

[0052]

[0053] The localization loss employs a combination of intersection-over-union (IoU) loss and time interval loss:

[0054] L loc =1-IoU(B pred B gt )+λ|[t s ,t e ]pred-[t s ,te ]gt|2,λ=0.5

[0055] The combined loss is:

[0056] L=αL cls +(1-α)L loc ,α∈[0,1]

[0057] F1. Optimize model parameters through backpropagation and iteratively update θ. v ,θ clip ,θ bert Etc., to ensure a synergistic improvement in behavior detection rate and positioning accuracy;

[0058] F2. Evaluate the behavior detection rate on the validation set. and positioning accuracy Where IoU is the intersection-union ratio between the predicted and the ground truth bounding boxes.

[0059] The beneficial effects of this invention are as follows: by actively discovering unknown behaviors and generating an interactive action category library through an action recognition module, and combining this with video grounding technology to achieve precise positioning, it breaks through the limitations of the traditional "passive response to known queries" and meets the "discover first, query later" requirement in open scenarios. Simultaneously, a bidirectional semantic alignment module is proposed to bridge the semantic gap between action recognition and video grounding, enhancing the deep fusion of visual and linguistic modalities and improving the robustness of behavior localization. Furthermore, a composite index of "behavior discovery rate - query response speed" is introduced to comprehensively measure the practical value of the system in open scenarios, providing a new evaluation benchmark for multimodal video understanding technology. This method also has broad application potential and can be transferred to scenarios such as intelligent security and industrial quality inspection, promoting the transition of multimodal technology from laboratory verification to large-scale application. Attached Figure Description

[0060] Figure 1 A schematic diagram of the overall process of an embodiment of the present invention;

[0061] Figure 2 Schematic diagram of the action recognition and behavior database generation process;

[0062] Figure 3 Semantic query and video grounding process diagram. Detailed Implementation

[0063] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0064] This embodiment uses monitoring video from a nature reserve as an example to verify the application effect of the present invention in wildlife behavior retrieval and localization, and achieves deep fusion of action recognition and video grounding based on the AnimalKingdom dataset.

[0065] like Figure 1As shown, this embodiment provides a method for retrieving and locating wildlife behavior videos based on deep learning, including the following steps:

[0066] Step 1, Data Preparation and Action Recognition Behavior Library Generation, aims to prepare high-quality training data, optimize the core model, and generate an interactive behavior category library through action recognition, providing a semantic foundation for subsequent video-based queries. By integrating the AnimalKingdom dataset, action categories and spatiotemporal segments are labeled. The TimeSformer and CLIP models are fine-tuned to optimize spatiotemporal feature extraction and visual-language mapping performance. The action recognition module extracts behavioral features from the input video to generate an interactive action category library, providing a semantic foundation for subsequent queries. Data preparation ensures high-quality labeling, model fine-tuning improves behavior recognition accuracy, and behavior library generation constructs a structured query interface through visual-language mapping and clustering algorithms.

[0067] Step 2, Semantic Query and Video Grounding, and Result Evaluation: This step triggers the video grounding module based on the semantic description selected by the user from the action category library or the natural language query input, locating the corresponding spatiotemporal segment and generating structured output. Precise positioning is achieved through bidirectional semantic alignment and spatiotemporal attention mechanisms. Combined with composite metrics to evaluate system performance, this ensures that the behavior detection rate, positioning accuracy, and query response speed meet the requirements of open scenarios.

[0068] Furthermore, such as Figure 2 The following steps are shown in the analysis of using YOLO and SAM to identify animal and environmental entities. Step one includes:

[0069] A0. Integrate the AnimalKingdom dataset, which contains approximately 5000 videos with a total duration of about 100 hours, covering 50 behavior categories. Each video is labeled with its behavior category, time interval, and spatial bounding box (format: ((x,y,w,h)), where ((x,y)) are normalized center coordinates, and ((w,h)) are width and height). LabelStudio was used to ensure category coverage was greater than 95%, and the bounding box IoU threshold was 0.8. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio.

[0070] A1. Fine-tune the TimeSformer model using the AnimalKingdom training set and the AdamW optimizer (learning rate (10)). -5 (batch size 16, training for 20 epochs), using cross-entropy loss to optimize spatiotemporal feature extraction; fine-tuning the CLIP model using contrastive learning loss, training for 10 epochs (learning rate (10) -6 Enhance visual-linguistic semantic alignment;

[0071] A2. Input surveillance video Frame sequence I is generated by sampling at 8 frames per second. t t=1 T and optical flow characteristics O t t=1 T (Extracted via RAFT). TimeSformer extracts spatiotemporal features. Predicting behavior probability using the classification head:

[0072]

[0073] Filter categories with a confidence level higher than 0.7 and generate C. selected =c i |P(c i |V)>0.7;

[0074] A3. Feature extraction using CLIP visual encoder Text encoder generates template features F t = CLIPtext(t i ;θclip), calculate semantic alignment:

[0075]

[0076] Retain action-text pairs with a similarity higher than 0.8;

[0077] A4. Features F of "Action-Text" pairs t Perform K-means clustering (K=50), merge redundant descriptions, and construct an action category library L=(c i ,t j ,F t It provides keyword search and t-SNE visualization capabilities through a web interface developed using Flask.

[0078] Step two includes:

[0079] B0, receive user natural language queries (Q) (such as "Find clips where a snow leopard is running") or select target behavior from the action category library;

[0080] B1. Input the query (Q) into the BERT model and extract the features (F). q =BERT(Q; θ) bert )∈R 768 ), dimensionality reduction through fully connected layers:

[0081] F′ q =W q F q +bq W q ∈R 512×768 ,b q ∈R 512

[0082] Apply Dropout (probability 0.3);

[0083] B2. Design a bidirectional semantic alignment module that utilizes an 8-head attention mechanism to enhance the association between queries and video features (F_v):

[0084]

[0085] B3. Construct a three-dimensional feature map (M∈R) based on spatiotemporal features (F_v). T×H / 4×W / 4 ), using spatiotemporal attention to calculate the matching score:

[0086] S st =Softmax(W s (M·F cross )+b s ),W s ∈R 512×1 ,b s ∈R

[0087] Output time interval [t] s ,t e ] and bounding box (B=(x,y,w,h));

[0088] B4. Generate structured output (JSON format), including behavior category, time range, and bounding box;

[0089] B5. Evaluation on the Animal Kingdom test set:

[0090] Behavioral detection rate:

[0091]

[0092] Positioning accuracy:

[0093]

[0094] Query response time: Average end-to-end response time is 1.2 seconds;

[0095] B6. Optimize the model using joint loss:

[0096]

[0097] Trained for 30 epochs using the AdamW optimizer (learning rate (10^{-4})).

[0098] In summary, this embodiment utilizes the AnimalKingdom dataset, employs the TimeSformer and CLIP models to achieve action recognition and visual-language mapping, generates an interactive action category library, and achieves precise localization through bidirectional semantic alignment and spatiotemporal attention mechanisms. It constructs a framework of "active perception → precise query," significantly improving behavior discovery rate and localization accuracy, with response speed meeting real-time requirements, and providing efficient support for ecological protection.

[0099] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without inventive effort. Therefore, those skilled in the art can make modifications or variations based on the concept of the present invention through logical analysis or limited experimentation, and all such modifications or variations should be included within the scope of protection defined by the claims.

Claims

1. A multimodal intelligent retrieval and localization method for wild animal movements, characterized in that, Includes the following steps: Step 1: Actively perceive various behaviors and actions in the video through the action recognition module, generate an interactive action category library, and provide a semantic basis for subsequent queries; Step 2: Based on the semantic description selected by the user from the action category library or the natural language query input, trigger the video grounding module to locate the corresponding spatiotemporal segment in the video; Step 3: By jointly optimizing the action recognition and video grounding modules, ensure the synergy between behavior discovery and localization, and generate a structured output that includes behavior category, time interval, and spatial region.

2. The method for retrieving and locating wildlife behavior videos based on deep learning as described in claim 1, characterized in that, Step one specifically includes: A0. Using a motion recognition model finely tuned on a wildlife video dataset, extract visual features from the input video stream to identify potential motion categories; A1. Based on the vision-language pre-trained model, visual features are mapped to the language semantic space to generate "action-text" candidate description pairs; A2. The action categories are normalized using a clustering algorithm to generate a structured action category library for interactive user queries.

3. The method for retrieving and locating wildlife behavior videos based on deep learning as described in claim 2, characterized in that, The A0 specifically includes: B0, For the input video V∈R T×H×W×3 Perform frame sampling and extract RGB frame sequence I. t t=1 T and optical flow characteristics O t t=1 T Where (T) is the number of frames and (H×W) is the frame resolution; B1. Using a pre-trained TimeSformer model, fine-tuned with a wildlife dataset, to extract the spatiotemporal features F of video frames. v =TimeSformer(V; θ) v )∈R T×512 , where θ v These are the fine-tuned model parameters; B2. Predict the action category probability P(c|V) using the classification head = Softmax(W c F v +b c ), where W c ∈R C×512 b c ∈R C C represents the number of action categories, filtering for behavior categories with a confidence level higher than 0.7: C selected =c i |(c i |)>0.7。 4. The method for retrieving and locating wildlife behavior videos based on deep learning as described in claim 2, characterized in that, A1 specifically includes: C0. Extracting high-level visual features F from video frames using a CLIP model visual encoder. clip = CLIPvisual(V; θclip)∈R T×512 , where θ clip These are model parameters; C1, Visual features F clip The input CLIP text encoder generates candidate text features F that match a predefined behavior description template. t = CLIPtext(t i ;θclip)∈R 512 ; C2. Calculate the semantic alignment between visual features and text features using cosine similarity: Retain action-text pairs with a similarity greater than 0.8: Pairs=(c i ,t j )|S(F clip ,F t )>0.8。 5. The method for retrieving and locating wildlife behavior videos based on deep learning as described in claim 2, characterized in that, A2 specifically includes: D0, Features F of the generated "action-text" pairs t K-means clustering was performed, and redundant descriptions were merged based on semantic similarity. The cluster centers were μ. k ∈R 512 Where k = 1, ..., K, and K is the number of clusters; D1. Construct an action category library L = (c i ,t j ,F t It stores the action category, the corresponding text description, and the feature vector; D2. Provide a web-based keyword search interface to present an action category library through keyword search for users to browse and select.

6. The method for retrieving and locating wildlife behavior videos based on deep learning as described in claim 1, characterized in that, Step two specifically includes: E0, receive a natural language query Q input by the user or select a target action description from the action category library; E1. Utilize a bidirectional semantic alignment module to perform cross-modal matching between language queries and video visual features, thereby enhancing the association weight between action-related words and visual features. E2. Through the spatiotemporal attention mechanism, locate and query matching spatiotemporal segments in the video stream, and output the specific time interval and spatial region where the behavior occurred.

7. The method for retrieving and locating wildlife behavior videos based on deep learning as described in claim 6, characterized in that, E1 specifically includes: F0. Encode the user-input language query Q using the BERT model and extract the text semantic features F. q =BERT(Q; θ) bert )∈R 768 ; F1. Dimensionality reduction to match visual features using fully connected layers: F′ q =W q F q +b q ,W q ∈R 512×768 ,b q ∈R 512 And apply Dropout (probability 0.3) to mitigate overfitting; F2. Design a bidirectional semantic alignment module to enhance language query and video feature integration through a multi-head attention mechanism. v The connection: F cross =Attention(Q,K,V),Q=W q F′ q ,K=W k F v ,V=W v F v Among them W q W k W v ∈R 512×512 It is a learnable projection matrix.

8. The method for retrieving and locating wildlife behavior videos based on deep learning as described in claim 6, characterized in that, E2 specifically includes: G0, Spatiotemporal features based on video frames F v Construct a 3D feature map M∈R T×H / 4×W / 4 Where H / 4 and W / 4 are the spatial downsampling resolutions; G1 utilizes a spatiotemporal attention mechanism, combined with cross-modal features F cross Calculate the matching score for each spatiotemporal region: S st =Softmax(W s (M·F cross )+b s ),W s ∈R 512×1 ,b s ∈R G2, Output the spatiotemporal segment with the highest matching score, including the time interval [t]. s ,t e The bounding box B = (x,y,w,h) is defined by the bounding box, where (x,y) are the normalized center coordinates and (w,h) are the bounding box dimensions.

9. The method for retrieving and locating wildlife behavior videos based on deep learning as described in claim 1, characterized in that, Step three specifically includes: H0. Design a joint loss function that fuses the classification loss L from action recognition. cls Positioning loss L with video grounding loc The classification loss uses cross-entropy: The localization loss employs a combination of intersection-over-union (IoU) loss and time interval loss: L loc =1-IoU(B pred ,B gt )+λ|[t s ,t e ]pred-[t s ,t e ]gt|2,λ=0.5 The combined loss is: [L=0.4Lcls+0.6L loc ] H1. Optimize model parameters through backpropagation and iteratively update θ. v ,θ clip ,θ bert This ensures a synergistic improvement in action detection rate (defined as the proportion of correctly identified action categories) and positioning accuracy (measured by IoU). H2. Generate structured output containing behavior categories, time intervals, and spatial regions, and store it in JSON format for user analysis or subsequent decision-making.

Images