Wild animal behavior automatic identification method based on knowledge graph

Abstract

The invention discloses a wild animal behavior automatic identification method based on a knowledge graph, and the method comprises the following steps: firstly, carrying out the target detection and precise segmentation of an image through employing YOLO and SAM technologies, and recognizing animals and environment entities; then, entity information and a relation thereof are extracted through two calculation branches of vision and characters; in the visual branch, feature extraction is carried out on a target position based on the OfficientNet-B4, and spatial features of animals and environments are obtained. In a character branch, after entity names are coded by using a BERT model, semantic features are extracted through text description. Afterwards, visual and character features are fused, the complex interaction relation between the visual and character features is captured through a multi-head attention mechanism, and the behavior relation between animals is generated; and finally, based on the identified entities and the relation information thereof, generating nodes and edges in the graph, and automatically constructing a knowledge graph for displaying wild animal behaviors.

Description

Technical Field

[0001] The present invention relates to the field of intelligent monitoring of wild animals, and specifically to an automatic recognition method of wild animal behaviors based on a knowledge graph, which is used for in-depth analysis of wild animal monitoring data. Background Art

[0002] In-depth research on the behavior of wild animals not only helps us understand their living conditions and interactions with the environment, but also provides a scientific basis for formulating more targeted protection and management strategies. By studying the daily activities and survival patterns of wild animals, experts can reduce interference from adverse factors and promote biodiversity and ecological balance.

[0003] However, traditional methods for studying wildlife behavior have many shortcomings. Direct observation and capture tagging require close proximity to animals, which may disturb their natural behavior and cause stress responses or behavioral changes. In addition, methods that rely on visual observation and manual recording are limited by the observer's field of vision and time, making it difficult to collect long-term, continuous data over a large area. These methods are time-consuming, labor-intensive, and inefficient, and are difficult to meet the needs of current animal research.

[0004] With the development of new generation information technology, researchers have begun to use equipment such as infrared trap cameras to continuously monitor and collect data on animal activities in the wild without disturbing the animals. After that, they made full use of intelligent technologies such as deep learning to process and analyze the image information of wild animals. However, these intelligent technologies mainly optimize the shallow recognition process of animals and cannot automatically identify the deep behavior of wild animals in the collected monitoring data. At the same time, due to the high requirements for professional knowledge such as computers, wildlife experts need the assistance of computer experts and cannot directly use these intelligent technologies. In other words, in the field of intelligent monitoring of wild animals, the automated in-depth understanding of animal behavior is still a challenge. Fortunately, in addition to intelligent technologies such as deep learning, the knowledge graph method can construct a relationship diagram that is easy for humans to understand, helping wildlife experts quickly understand the behavioral relationship between animals. Therefore, it is urgent to fully combine the results of intelligent monitoring technologies such as deep learning, give full play to the advantages of knowledge graphs that are easy for human experts to read and understand, and design a method for automatic identification of wild animal behavior based on knowledge graphs, so that researchers in the field of zoology can carry out zoological research based on the new generation of information technology. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to provide an automatic identification method for wild animal behavior based on knowledge graph, which can provide wild animal experts with high-quality animal behavior and its relationship results by automatically identifying animal targets and environmental categories and utilizing the characteristics of knowledge graph that are easy for humans to understand.

[0006] To achieve the above object, the present invention provides a method for automatically identifying wild animal behaviors based on a knowledge graph, which is characterized by comprising the following steps:

[0007] Step 1: Use YOLO and SAM to identify animals and environmental entities;

[0008] Step 2: Extract the relationship between entities and generate a knowledge graph based on the fusion model of visual and text features.

[0009] Further, step one specifically includes:

[0010] A0. Using the YOLO detection model fine-tuned on the wildlife dataset to efficiently identify the bounding box set of animals and environmental target objects from the images taken by the original infrared camera And determine the entity class name of the animal or environment T i . Where N is the number of detected objects, each bounding box B i The center coordinates are normalized to the range [0,1] and size Indicates that

[0011] A1. Use LoRA (Low-Rank Adaptation of Large Language Models) technology to fine-tune the segmentation model SAM (Segment Anything Mode) on the wildlife dataset to obtain a SAM segmentation model suitable for wildlife data;

[0012] A2. Generate positive and negative cue points to input into SAM, with each bounding box B i The segmentation masks of the animal object and the environmental object are obtained based on the accurate segmentation masks of the animal entity and the environmental entity;

[0013] A3. Screen the obtained segmentation masks to obtain high-quality masks for representing animal entities and environmental entities.

[0014] Furthermore, step A2 in step 1 specifically includes:

[0015] B1. In each bounding box B containing the environment entity i K positive cue points are randomly generated as the input cue of SAM. For the i-th bounding box, the k-th positive cue point (x i,k ,y i,k ) is:

[0016] k=1,2,…,K,

[0017] in, represents a uniform distribution on the interval [a,b]. The labels of all positive prompt points are set to l i,k =1;

[0018] B2. Randomly select K' labels from the segmented animal entity mask and set them as l i,k = 0 as negative prompt points to solve the problem that environmental entities are usually irregular and have a high degree of overlap with animal entities;

[0019] B3, image I, cue point set and bounding box collection Input into the fine-tuned SAM to generate segmentation masks of all animal entities and environmental entities in image I:

[0020]

[0021] Furthermore, in step 1, A3 specifically includes:

[0022] C0, by calculating the mask M in the mask set j and the corresponding bounding box B i The intersection over union (IoU) of the image is used to discard masks of poor quality and retain high-quality masks for knowledge graph generation:

[0023]

[0024] C1, discard IoU (M j ,B i )<0.5, and obtain a high-quality mask set Used to represent animal entities and environmental entities.

[0025] Furthermore, step 2 specifically includes:

[0026] D0. For any two entities, extract high-level image semantic features based on visual models and masks;

[0027] D1. For any two entities, extract high-level text semantic features based on the language model and entity category names;

[0028] D2. After integrating high-level image semantic features and high-level text semantic features, a knowledge graph containing animal entities and environmental entities is generated.

[0029] Furthermore, in step 2, D0 specifically includes:

[0030] E1. Use the pre-trained efficient convolutional neural network EfficientNet-B4 network and the Squeeze-and-Excitation (SE) module that can enhance the inter-channel feature expression as the backbone model to extract images Feature map Where C = 1792;

[0031] E2. Collect high-quality masks After pixel multiplication with feature map F, we get feature map f that retains entity information. v ;

[0032] E3, use a fully connected layer including two layers of ReLU activation layers, group normalization (GN) and Dropout layers to extract high-level image semantic features h from visual features v :

[0033] h v =Dropout(GN(ReLU(W2·Dropout(GN(ReLU(W1f v +b1)))+b2))),

[0034] in, is the weight matrix of the fully connected layer, h v is the final mapped 512-dimensional vector.

[0035] Furthermore, step D1 in step 2 specifically includes:

[0036] F1. Change the entity class name of the entity animal or environment to T i Input into the BERT (Bidirectional Encoder Representations from Transformers) model to obtain the semantic feature vector

[0037] F2, use a fully connected layer to transform the semantic feature vector f l Dimensionality is reduced to be consistent with high-level image semantic features, and a drop-out mechanism is added to reduce overfitting to obtain high-level semantic text features:

[0038] h l = Dropout(GN(ReLU(W l f l +b l ))),

[0039] in, is the weight matrix of the fully connected layer.

[0040] Furthermore, step D2 in step 2 specifically includes:

[0041] G1. For any two entities, the high-level image semantic features h v As a query, the high-level text semantic feature hl As the key and value, the multi-head attention mechanism is used to obtain the fused feature h fused =MultiHeadAttention(h v ,h l ,h l );

[0042] G2, the fusion feature h fused After the softmax layer, we get the probability distribution of any two entity relationships, softmax(h fused ), the relationship type with probability greater than 0.5 is retained as the relationship between the two entities. If there is no relationship type greater than 0.5, it indicates that there is no relationship between the two entities.

[0043] G3. Store entities and inter-entity relationships and probability values ​​in a multivariate array, and use knowledge graph visualization technology to form an animal behavior knowledge graph that is easy for human animal experts to understand.

[0044] The beneficial effect of the present invention is that the present invention can automatically identify wild animals and environmental entities from data collected by commonly used wildlife monitoring equipment such as trap cameras, analyze animal behavior, and visualize and generate corresponding knowledge graphs for easy understanding and application by wildlife experts. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the process of the first embodiment of the present invention;

[0046] Figure 2 This is a flow chart of using YOLO and SAM to identify animals and environmental entities;

[0047] Figure 3 It is a schematic diagram of the process of using a fusion model of visual and text features to extract the relationship between entities and generate a knowledge graph; DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0049] This embodiment is to identify the behavior of wild animals from pictures taken by trap cameras and generate a knowledge graph.

[0050] like Figure 1 As shown, this embodiment provides a method for automatically identifying wild animal behaviors based on a knowledge graph, comprising the following steps:

[0051] Step 1: Integrate the two datasets LoTE-animal and ATRW to form an integrated dataset. The LabelImg tool is used to annotate the animals, environmental entities and their corresponding relationships. The resulting dataset contains the interactive relationship between the target animals and the target environment. In order to ensure the reasonable distribution of the data, the dataset is divided according to a certain ratio. The two-stage method of YOLO and SAM is used to accurately identify and segment animals and environmental entities from images taken by infrared cameras.

[0052] Step 2: Input the high-quality mask set obtained in step 1 and the recognized entity pair names into the fusion model of visual and text features respectively, extract the relationship between entities and generate a knowledge graph based on the fusion model of visual and text features. In the visual feature extraction module, the pre-trained EfficientNet-B4 is used as the backbone network to extract rich high-level semantic features from the image. In the text feature extraction module, the pre-trained BERT model is used to obtain high-level text semantic features of object pairs. In the feature fusion and relationship prediction module, a multi-head attention mechanism is introduced to fuse the above visual and text features. Finally, the relationship is predicted and a knowledge graph is constructed based on the entities and relationships.

[0053] Furthermore, if Figure 2 The following example shows how to use YOLO and SAM to identify animals and environmental entities. Step 1 includes:

[0054] A0, based on the integration of LoTE-animal and ATRW datasets, to form an integrated dataset;

[0055] A1. LabelImg tool is used to label the animals, environmental entities and their corresponding relationships on the integrated data set. The resulting data set contains the interactive relationship between the target animals and the target environment.

[0056] A2. The data set is randomly divided into training set, validation set and test set in a ratio of 8:1:1;

[0057] A3. Fine-tune YOLO on the processed wildlife dataset.

[0058] A4. Use the YOLO detection model fine-tuned on the wildlife dataset to efficiently identify the bounding box set of animals and environmental target objects from the images taken by the original infrared camera And determine the entity class name of the animal or environment T i . Where N is the number of detected objects, each bounding box B i The center coordinates are normalized to the range [0,1] and size Indicates that

[0059] A5. Use LoRA technology to fine-tune the segmentation model SAM on the wildlife data set to obtain a segmentation model suitable for wildlife data;

[0060] A6, generate positive and negative cue points and input them into SAM to obtain the segmentation masks of animal entities and environmental entities;

[0061] A7. Filter the obtained segmentation mask to obtain a high-quality mask for input into a fusion model based on visual and text features.

[0062] Furthermore, the specific steps of generating positive and negative cue points as described in A6 and inputting them into SAM to obtain the segmentation masks of the animal entity and the environment entity are as follows:

[0063] B1. In each bounding box B containing the environment entity i K positive cue points are randomly generated as the input cue of SAM. For the i-th bounding box, the k-th positive cue point (x i,k ,y i,k ) is:

[0064] k=1,2,…,K,

[0065] in, represents a uniform distribution on the interval [a,b]. The labels of all positive prompt points are set to l i,k =1;

[0066] B2. Randomly select K' labels from the segmented animal entity mask and set them as l i,k = 0 as negative prompt points to solve the problem that environmental entities are usually irregular and have a high degree of overlap with animal entities;

[0067] B3, image I, cue point set and bounding box collection Input into the fine-tuned SAM to generate segmentation masks of all animal entities and environmental entities in image I:

[0068]

[0069] Furthermore, the specific steps of screening the obtained segmentation mask described in A7 to obtain a high-quality mask for input into the fusion model based on visual and text features are as follows:

[0070] C0, by calculating the mask M in the mask set j and the corresponding bounding box B iThe intersection over union (IoU) of the image is used to discard masks of poor quality and retain high-quality masks for knowledge graph generation:

[0071]

[0072] C1, discard IoU (M j ,B i )<0.5, and obtain a high-quality mask set As input to the vision model.

[0073] Furthermore, if Figure 3 As shown in the figure, the fusion model based on visual and text features is used to extract the relationship between entities and generate a knowledge graph. Step 2 includes:

[0074] D0. Input the image and high-quality mask set into the visual model;

[0075] D1. For any two entities in the image, concatenate the entity category names and input them into the text model;

[0076] D2. Integrate high-level image semantic features and high-level text semantic features to generate a knowledge graph containing animal entities and environmental entities.

[0077] Further, the specific steps of inputting the image and the high-quality mask set into the visual model described in D1 are as follows:

[0078] E0, use the pre-trained efficient convolutional neural network EfficientNet-B4 network and the Squeeze-and-Excitation (SE) module that can enhance the inter-channel feature expression as the backbone model to extract images The feature map F∈R C×H′×W′ , where C = 1792;

[0079] E1. Collect high-quality masks After pixel multiplication with feature map F, we get feature map f that retains entity information. v ;

[0080] E2, use a fully connected layer including two layers of ReLU activation layers, group normalization (GN) and Dropout layers to extract high-level image semantic features h from visual features v :

[0081] h v =Dropout(GN(ReLU(W2·Dropout(GN(ReLU(W1f v +b1)))+b2))),

[0082] in, is the weight matrix of the fully connected layer, h v is the final mapped 512-dimensional vector;

[0083] E3. Map the visual features into a 512-dimensional vector h v , as the query input of the fusion part.

[0084] Furthermore, the specific steps of concatenating the entity category names and inputting them into the text model for any two entities in the image described in D1 are as follows:

[0085] F0, entity pair text T that concatenates the entity category names of entity animals or environments i Input into the BERT model to obtain the semantic feature vector

[0086] F1, use a fully connected layer to transform the semantic feature vector f l Dimensionality is reduced to be consistent with high-level image semantic features, and a drop-out mechanism is added to reduce overfitting to obtain high-level semantic text features:

[0087] h l = Dropout(GN(ReLU(W l f l +b l ))),

[0088] in, is the weight matrix of the fully connected layer.

[0089] F2, map text features into a 512-dimensional vector h l , as the key and value input of the fusion part.

[0090] Furthermore, the specific steps of integrating high-level image semantic features and high-level text semantic features described in D2 to generate a knowledge graph containing animal entities and environmental entities are as follows:

[0091] G0, for any two entities, the high-level image semantic features h v As a query, the high-level text semantic feature h l As the key and value, the multi-head attention mechanism is used to obtain the fused feature h fused =MultiHeadAttention(h v ,h l ,h l );

[0092] G1, fusion feature h fusedAfter the softmax layer, we get the probability distribution of any two entity relationships, softmax(h fused ), the relationship type with probability greater than 0.5 is retained as the relationship between the two entities. If there is no relationship type greater than 0.5, it indicates that there is no relationship between the two entities.

[0093] G2. Store entities and their relationships and probability values ​​in a multivariate array, and use knowledge graph visualization technology to form an animal behavior knowledge graph that is easy for human animal experts to understand. In the knowledge graph, entities serve as nodes and relationships as edges.

[0094] In summary, this embodiment uses YOLO and SAM to identify animal and environmental entities from pictures taken by trap cameras, extracts the relationship between entities based on the fusion model of visual and text features, and generates a knowledge graph, aiming to achieve automatic recognition and analysis of wild animal behavior by combining deep learning and knowledge graph technology. The core idea of ​​this method is to construct a relationship map between animal and environmental entities through the fusion of visual and text information, so as to deeply understand the behavioral habits of wild animals.

[0095] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for automatically identifying wild animal behaviors based on knowledge graph, characterized in that: The steps include: Step 1: Use YOLO and SAM to identify animals and environmental entities; Step 2: Extract the relationship between entities and generate a knowledge graph based on the fusion model of visual and text features.

2. The method for automatically identifying wild animal behaviors based on knowledge graph according to claim 1, characterized in that: The step 1 specifically includes: A0. Using the YOLO detection model fine-tuned on the wildlife dataset to efficiently identify the bounding box set of animals and environmental target objects from the images taken by the original infrared camera And determine the entity class name of the animal or environment T i . Where N is the number of detected objects, each bounding box B i The center coordinates are normalized to the range [0,1] and size Indicates that A1. Use LoRA (Low-Rank Adaptation of Large Language Models) technology to fine-tune the segmentation model SAM (Segment Anything Mode) on the wildlife dataset to obtain a SAM segmentation model suitable for wildlife data; A2. Generate positive and negative cue points to input into SAM, with each bounding box B i The segmentation masks of the animal object and the environmental object are obtained based on the accurate segmentation masks of the animal entity and the environmental entity; A3. Screen the obtained segmentation masks to obtain high-quality masks for representing animal entities and environmental entities.

3. The method for automatically identifying wild animal behaviors based on knowledge graph according to claim 2, characterized in that: The specific steps of generating positive and negative cue points as described in A2 and inputting them into SAM to obtain the segmentation masks of the animal object and the environmental object are as follows: B1. In each bounding box B containing the environment entity i K positive cue points are randomly generated as the input cue of SAM. For the i-th bounding box, the k-th positive cue point (x i,k ,y i,k ) is: in, represents a uniform distribution on the interval [a,b]. The labels of all positive prompt points are set to l i,k =1; B2. Randomly select K' labels from the segmented animal entity mask and set them as l i,k = 0 as negative prompt points to solve the problem that environmental entities are usually irregular and have a high degree of overlap with animal entities; B3, image I, cue point set and bounding box collection Input into the fine-tuned SAM to generate segmentation masks of all animal entities and environmental entities in image I:

4. The method for automatically identifying wild animal behaviors based on knowledge graph according to claim 2, characterized in that: The specific steps of screening the obtained segmentation mask described in A3 to obtain a high-quality mask are as follows: C0, by calculating the mask M in the mask set j and the corresponding bounding box B i The intersection over union (IoU) of the image is used to discard masks of poor quality and retain high-quality masks for knowledge graph generation: C1, discard IoU (M j ,B i )<0.5, and obtain a high-quality mask set Used to represent animal entities and environmental entities.

5. The method for automatically identifying wild animal behaviors based on knowledge graph according to claim 1, characterized in that: The step 2 specifically includes: D0. For any two entities, extract high-level image semantic features based on visual models and masks; D1. For any two entities, extract high-level text semantic features based on the language model and entity category names; D2. After integrating high-level image semantic features and high-level text semantic features, a knowledge graph containing animal entities and environmental entities is generated.

6. The method for automatically identifying wild animal behaviors based on knowledge graph according to claim 5, characterized in that: The specific steps of extracting high-level image semantic features based on visual models and masks for any two entities described in D0 are as follows: E1. Use the pre-trained efficient convolutional neural network EfficientNet-B4 network and the Squeeze-and-Excitation (SE) module that can enhance the inter-channel feature expression as the backbone model to extract images The feature map F∈R C×H′×W′ , where C = 1792; E2. Collect high-quality masks After pixel multiplication with feature map F, we get feature map f that retains entity information. v ; E3, use a fully connected layer including two layers of ReLU activation layers, group normalization (GN) and Dropout layers to extract high-level image semantic features h from visual features v : in, is the weight matrix of the fully connected layer, h v is the final mapped 512-dimensional vector.

7. The method for automatically identifying wild animal behaviors based on knowledge graph according to claim 5, characterized in that: The specific steps for extracting high-level text semantic features based on the language model and entity category names for any two entities described in D1 are as follows: F1. Change the entity class name of the entity animal or environment to T i Input into the BERT (Bidirectional Encoder Representations from Transformers) model to obtain the semantic feature vector F2, use a fully connected layer to transform the semantic feature vector f l Dimensionality is reduced to be consistent with high-level image semantic features, and a drop-out mechanism is added to reduce overfitting to obtain high-level semantic text features: in, is the weight matrix of the fully connected layer.

8. The method for automatically identifying wild animal behaviors based on knowledge graph according to claim 5, characterized in that: The specific steps of generating a knowledge graph containing animal entities and environmental entities by fusing high-level image semantic features and high-level text semantic features as described in D2 are as follows: G1. For any two entities, the high-level image semantic features h v As a query, the high-level text semantic feature h l As the key and value, the multi-head attention mechanism is used to obtain the fused feature h fused =MultiHeadAttention(h v ,h l ,h l ); G2, the fusion feature h fused After the softmax layer, we get the probability distribution of any two entity relationships, softmax(h fused ), the relationship type with probability greater than 0.5 is retained as the relationship between the two entities. If there is no relationship type with probability greater than 0.5, it indicates that there is no relationship between the two entities; G3. Store entities and inter-entity relationships and probability values ​​in a multivariate array, and use knowledge graph visualization technology to form an animal behavior knowledge graph that is easy for human animal experts to understand.

Images