A method for predicting mating period of giant panda based on multi-modal behavior information

Abstract

The present application belongs to the field of endangered animal protection, and particularly relates to a giant panda mating period prediction method based on multi-modal behavior information. The present application provides the giant panda mating period prediction method based on multi-modal behavior information, uses channel fusion plus attention mechanism to extract the time and space information contained in the video data, strengthens the interaction of time information and space information in the video input, uses audio as a supplement to the video data, comprehensively analyzes the action behavior and roaring behavior of the giant panda, and improves the accuracy of predicting the estrus stage change of the giant panda through behavior change, and the time sequence prediction model proposed is tested on the behavior change sequence data to obtain a mean square error of 0.400 and a root mean square error of 0.472, which has obvious performance advantages compared with other time sequence prediction methods.

Description

Technical Field

[0001] This invention belongs to the field of endangered animal protection, specifically involving a method for predicting the mating period of giant pandas based on multimodal behavioral information. Background Technology

[0002] Giant pandas have always been a focus of global animal conservation research due to their extraordinary cultural, scientific, and economic value. In giant panda breeding and conservation efforts, choosing the right mating time is crucial. Female giant pandas only have one ovulation opportunity per year, and their acceptable natural mating time is extremely short, sometimes only a few hours. Failure to choose the appropriate mating time will hinder successful mating and may even lead to fierce fighting and animal injuries or deaths.

[0003] Theoretically, the optimal mating time for female giant pandas is 10-28 hours after the surge in urinary estrogen and before the peak of luteinizing hormone (LH) levels. Therefore, currently, experts determine mating timing by observing changes in the appearance of the panda's genitals and hormone levels. This requires experts to constantly monitor panda behavior, regularly collect urine or blood samples from pandas in estrus, and use real-time tracking of hormone level fluctuations in blood or urine to determine if the panda is suitable for mating. This method heavily relies on panda excrement; reducing urine sample contamination requires significant human resources for monitoring and inevitably disrupts the pandas' daily lives. Furthermore, the delay in hormone testing makes selecting the appropriate mating time difficult. Overall, current methods for predicting panda mating times cannot provide relatively accurate predictions with minimal human intervention and without disrupting the pandas' daily lives. Summary of the Invention

[0004] This invention provides a method for predicting the mating season of giant pandas based on multimodal behavioral information. Using multimodal video recorded by a camera as input, a deep learning model is used to obtain behavioral data of giant pandas over a period of time, and then the accurate mating season is predicted. This method can reduce manual workload and interference with the daily lives of giant pandas, thus promoting the development of giant panda conservation efforts.

[0005] The technical solution of this invention is:

[0006] A method for predicting giant panda mating time based on multimodal behavioral information, such as Figure 1 As shown, it includes the following steps:

[0007] S1. Define two modalities for giant panda behavior: audio and video. Collect videos of female giant pandas during their estrus cycle and label them modally to obtain a multimodal dataset. Specifically:

[0008] S11. Define giant panda behavior in different modalities, including giant panda action behavior in video modality and giant panda roaring behavior in sound modality;

[0009] S12. Separate the video stream and audio stream within the recorded video to obtain video stream data consisting of image frames and audio stream data consisting of audio segments. For the video stream data, label the spatial location and behavioral tags of the giant panda in each frame; for the audio stream data, label the type of roar to which each audio segment belongs.

[0010] S2. Design a multimodal behavior feature extraction module, such as... Figure 2 As shown, it includes:

[0011] S21. Video stream: A dual-stream network is used. Temporal information is extracted from consecutive input video frames using a 3D CNN, and spatial information from input keyframes is extracted using a 2D CNN.

[0012] F 2d =φ 2d (I k ), F 3d =φ 3d (I s )

[0013] Transformation φ 2d (·) and φ 3d (·) represent 2D CNN and 3D CNN respectively, I k I represents the keyframe of the video input. s This represents the sequence of frames in the video input. Indicates time characteristics, Represents spatial characteristics. B represents the batch size, C... s and C t H and W represent the channel dimensions of spatial and temporal features, respectively, and H and W represent the length and width of the feature map. Channel fusion will then be performed.

[0014]

[0015] conv1(·) represents a convolutional layer with a kernel size of 1x1. conv3(·) represents a convolutional layer with a kernel size of 3x3. The channel fusion refers to combining the temporal features F... 3d and spatial features F 2d The concatenation is performed along the channel dimension, and then processed using a 1x1 convolution and a 3x3 convolution to output the fused features. After channel fusion, the fused features are first processed. Perform a flattening operation to obtain a vector. Then let F′ pass through the self-attention layer. The formula for calculating the self-attention layer is as follows:

[0016] Q = K = V = F'

[0017] W = softmax(QK) T )

[0018]

[0019] F″=F+αF a

[0020] In this model, Q represents the Query, K the Key, V the Value, and W the Weight Factor, which represents the similarity between channels in F′, distinguished by the channel itself. α determines how much attention information is retained; it is initialized to 0 and is a learnable parameter. After processing through the channel-based attention mechanism, F″ better integrates the two different feature information outputs from 3D CNN and 2D CNN. F″ obtains the classification result of panda behavior in the video modality through a classification head.

[0021] S22, Sound Stream: First, a one-dimensional convolutional layer is used to fuse the input audio information and unify the dimension of the sound wave vector. Then, a continuous multi-scale feature extraction module is used to process the multi-scale features contained in the audio. Finally, context-related pooling and AAM (Additive Angular Margin)-softmax are used to obtain the accurate type of giant panda roar behavior.

[0022] S23. The classification results obtained from the combined video stream and audio stream are used to obtain the multimodal behavior embedding vector.

[0023] S3. Train a multimodal behavior feature extraction module based on a multimodal dataset to detect panda videos within a certain time window to obtain behavior change data. Specifically, if the observation window is T days, the obtained panda behavior change vector is:

[0024] X = [x1, x2, ... x] t ..., x T ]

[0025] in This is the multimodal behavior embedding vector obtained in the previous step, where N is the number of defined giant panda behavior categories, and x... t The value of each element is the ratio of the duration of each line to the duration of the input video.

[0026] S4. Design the time series forecasting module. For example... Figure 3 As shown, for the input giant panda behavior change vector X, the behavior change information is first extracted through a one-dimensional convolutional layer, then passed through three LSTM layers, and finally through a self-attention operation and a classification head to obtain the classification result. The calculation formula for the LSTM layer is as follows:

[0027] f t =σ(W f ·[h t-1 ;x t ]+b f )

[0028] i t =σ(W i ·[h t-1 ;x t ]+b i )

[0029]

[0030]

[0031] o t =σ(W o ·[h t-1 ;x t ]+b o )

[0032] h t =o t *tanh(C t )

[0033] f t i t , C t o t Let W represent the forget gate, input gate, candidate memory, memory gate, and output gate, respectively. Let σ(·) represent the sigmoid function. f W i W C W o Let b represent the forget gate weight matrix, input gate weight matrix, memory state weight matrix, and output gate weight matrix of the LSTM unit, respectively. f b i b C b o h is the corresponding bias matrix. t-1 h is the output of the hidden layer at the previous time step. t is the hidden layer output at the current time step, and tanh(·) is the hyperbolic tangent function.

[0034] S5. Train the time series prediction module based on behavioral change data to obtain the prediction results for the mating period of giant pandas.

[0035] This invention uses the time between the last day of the observation window and the actual mating date as the label for predicting the mating period, employing mean absolute error (MAE), root mean square error (RMSE), and R0. 2 The coefficient of determination is used to evaluate prediction effectiveness, and its calculation formula is as follows:

[0036]

[0037]

[0038]

[0039] in y represents the model's predicted value. i Indicates the true label, This represents the average value of the tags. MAE and RMSE reflect the difference between the model-predicted mating date and the actual mating date, R0. 2 It reflects the goodness of fit of the model.

[0040] The beneficial effects of this invention are as follows: This invention provides a method for predicting the mating period of giant pandas based on multimodal behavior recognition and time series analysis. Incorporating audio information into the consideration of behavioral change information can improve the accuracy and robustness of prediction. For the video behavior extraction module: 3D CNN is used to extract temporal information, and 2D CNN is used to extract spatial information, achieving a balance between accuracy and real-time performance. Channel fusion combined with a self-attention mechanism effectively integrates the temporal and spatial information contained in the video modalities, improving the accuracy of video behavior prediction. This invention requires only a small amount of manual labor, can operate around the clock, and has minimal interference with the daily life of giant pandas. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0042] Figure 2 This is a schematic diagram of the multimodal behavior feature extraction module.

[0043] Figure 3 A schematic diagram of the time series prediction module; Detailed Implementation

[0044] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0045] refer to Figure 1 In one embodiment of the present invention, the present invention provides a method for predicting the mating period of giant pandas based on multimodal behavioral information, comprising the following steps:

[0046] Step 1: Define two modalities for giant panda behavior: audio and video. Collect videos of female giant pandas in estrus, and annotate the videos modally to obtain a multimodal dataset. This can be broken down into the following steps:

[0047] Step 1.1: Previous studies on behavioral changes and activity rhythms during the estrus period of captive giant pandas generally categorized their external behaviors into six types: feeding, adaptation, resting, exercise, exploration, and estrus. This invention, based on the characteristics of model-based identification of specific behaviors, identifies sixteen behaviors representing the above six types of giant pandas, and further categorizes them into video modalities and auditory modalities. Specific behavioral characteristics are described in the table below:

[0048] Table 1. Giant panda behaviors identified by this invention.

[0049] walk The forelimbs and hindlimbs alternated side to side, and the body moved slowly. video Climbing The body moves along the tree trunk and other supports by alternating the movement of its forelimbs and hindlimbs. video sit For a period of time, the buttocks are in direct contact with the ground, the hind limbs extend forward, and the front... video lie For a period of time, the chest and abdomen are facing downwards, the limbs are pressed under the body, or the forelimbs are forward. video squat For a period of time, stand in a semi-standing position with hips lowered and close to the ground, and grasp with forelimbs. video lie For a period of time, the back or one side of the body is in contact with the ground. video rub Rub the surface close to the wall, protrusions, or the ground in an arc or straight line. video Rolling rolling around on the ground video Drinking water Drink water by sucking it from the water bowl (jar). video feeding Use their mouth and paws to handle and swallow food. video licking Open your mouth and stick out your tongue to lick other objects video rubbing against the shady side With the perianal gland area close to the wall, protrusions, or ground, make an arc or... video Tail The individual stands upright and naturally raises its tail. video Urine and feces marking Individuals urinate or defecate in various postures. video Meow The individual emitted a sound similar to a sheep's bellow. sound Birdsong The individual emitted a sound similar to a bird's cry. sound

[0050] Step 1.2: Set up cameras outside the enclosure to record data, and separate the recorded video and audio. Divide the separated video into frames and label the spatial location and behavior of the pandas within each frame. Divide the separated audio into segments and label the type of panda roaring behavior in each audio segment.

[0051] Step 2: Design a multimodal behavior feature extraction module, such as... Figure 2 As shown, the video stream first extracts temporal information from consecutive input video frames using a 3D CNN, then extracts spatial information from input keyframes using a 2D CNN, and then fuses the temporal and spatial information using channel fusion and attention mechanisms. Finally, an accurate panda action classification result is obtained through a classification head. The audio stream first uses a one-dimensional convolutional layer to fuse the input audio information and unify the dimension of the sound wave vector. Then, a continuous multi-scale feature extraction module processes the multi-scale information contained in the audio, and finally, context-related pooling and AAM softmax are used to obtain an accurate classification of panda roaring behavior.

[0052] Step 3: Train a multimodal behavioral feature extraction module based on the multimodal dataset to detect estrus period videos and obtain giant panda behavioral change data. Specifically, if the observation window is T days, the obtained giant panda behavioral change vector is:

[0053] X = [x1, x2, ... x] t ..., x T ]

[0054] in This is the multimodal behavior embedding vector obtained in the previous step, where N is the number of defined giant panda behavior categories, and x... t The value of each element is the ratio of the duration of each line to the duration of the input video.

[0055] Step 4: Design the time series forecasting module, such as... Figure 3 As shown, for the input giant panda behavior change vector, the behavior change information is first extracted through a one-dimensional convolutional layer, then passed through three LSTM layers, and finally through a self-attention operation and a classification head to obtain the prediction result.

[0056] Step 5: Train the time series prediction module based on the behavioral change data, and analyze the time series behavioral change data to obtain the prediction results.

[0057] Data were collected from eight female giant pandas in estrus at the Chengdu Research Base of Giant Panda Breeding in 2020 and 2021. An observation window of 5 days was defined, and the predicted mating date was labeled as the difference in the number of days between the last day of the observation window and the actual mating date chosen in previous years. The prediction performance of this invention was compared with that of three other methods as follows:

[0058] Table 2 compares the prediction method used with the other three methods.

[0059] MLP 0.579 0.727 0.063 Transformer 0.525 0.671 0.186 LSTM 0.510 0.671 0.303 ours 0.400 0.472 0.878

[0060] The results show that, thanks to the attention mechanism and the ability of LSTM to extract time series information, the mating time prediction method proposed in this invention has smaller mean square error, root mean square error and higher goodness of fit.

Claims

1. A method for predicting the mating period of giant pandas based on multimodal behavioral information, characterized in that, Includes the following steps: S1. Define two modalities of giant panda behavior: sound and video. Collect videos of female giant pandas in estrus and label them by modality to obtain a multimodal dataset. Specifically, it includes: S11. Define giant panda behavior in different modalities, including giant panda action behavior in video modality and giant panda roaring behavior in sound modality; S12. Separate the video stream and audio stream within the recorded video to obtain video stream data consisting of image frames and audio stream data consisting of audio segments; for the video stream data, label the spatial location and behavioral tags of the giant panda in each frame; for the audio stream data, label the type of roar to which each audio segment belongs. S2. Design a multimodal behavior feature extraction module, including: S21. For the video stream, a dual-stream network is used to extract the temporal information from the consecutive input video frames using a 3D CNN, and the spatial information from the input keyframes using a 2D CNN. ， Transformation and These represent 2D CNN and 3D CNN, respectively. This represents the keyframe of the video input. This represents the sequence of frames in the video input. Indicates time characteristics, B represents spatial characteristics, and B represents the batch size. H and W represent the channel dimensions of spatial and temporal features, respectively, and represent the length and width of the feature map; channel fusion is then performed. ， This indicates a convolutional layer with a kernel size of 1x1. This indicates a convolutional layer with a kernel size of 3x3; channel fusion refers to the fusion of temporal features. and spatial features The concatenation is performed along the channel dimension, and then processed using a 1x1 convolution and a 3x3 convolution to output the fused features. After channel fusion, the fused features are first processed. Perform a flattening operation to obtain a vector Then let After passing through the self-attention layer, the formula for calculating the self-attention layer is as follows: ， ， ， ， Where Q stands for Query, K for Key, and V for Value. The weighting factor represents the distinction based on the channel. The degree of similarity between the various channels, Used to determine how much attentional information is retained, it is initialized to 0 and is a learnable parameter; after processing by the attention mechanism on the channel. This better integrates the two different feature information outputs from 3D CNN and 2D CNN. The classification results of giant panda behavior in the video modality are obtained through the classification head; S22. For the sound stream, firstly, a one-dimensional convolutional layer is used to fuse the input audio information and unify the dimension of the sound wave vector. Then, the multi-scale features contained in the audio are processed through continuous multi-scale feature extraction mode. Finally, the accurate panda roaring behavior type is obtained through context-related pooling and AAM-softmax. S23. The classification results obtained from the combined video stream and audio stream are used to obtain the multimodal behavior embedding vector; S3. Train a multimodal behavior feature extraction module based on a multimodal dataset to detect panda videos within a certain time window to obtain behavior change data; specifically, define the observation window as T days, then the obtained panda behavior change vector is: ， in This is the multimodal behavior embedding vector obtained in the previous step, where N is the number of defined giant panda behavior categories. The value of each element is the ratio of the duration of each line to the duration of the input video; S4. Design a time series prediction module for the input giant panda behavior change vector. First, a one-dimensional convolutional layer extracts behavioral change information. Then, it passes through three LSTM layers, and finally, a self-attention operation and a classification head are used to obtain the classification result. The calculation formula for the LSTM layer is as follows: ， ， ， ， ， ， , , , , These represent the forget gate, input gate, candidate memory, memory gate, and output gate, respectively. This refers to the sigmoid function. These represent the forget gate weight matrix, input gate weight matrix, memory state weight matrix, and output gate weight matrix of an LSTM unit, respectively. , , , For the corresponding bias matrix, This is the output of the hidden layer from the previous time step. This is the output of the hidden layer at the current moment. It is the hyperbolic tangent function; S5. Train the time series prediction module based on behavioral change data to obtain the prediction results for the mating period of giant pandas.

Images