A bird flight video key point extraction method fusing a YOLO network and an LSTM network

Abstract

The application discloses a kind of fusion YOLO network and bird flight video key point extraction method of LSTM network, belong to image data processing field, it is in to solve the problem of existing method when using YOLO network technology to extract bird skeleton key point, traditional method relies on single frame image, lack of time series information association, difficult to capture its dynamic change process completely, leading to insufficient posture analysis precision.It includes the following steps: constructing multi-view bird dataset;Get enhanced bird dataset;Get pre-training model;Get second post-training model;Predict the continuous flight video of the bird to be analyzed using the second post-training model;Time series modeling correction.Compared with the prior art, the application effectively improves the model generalization ability by two-stage training strategy and network structure optimization, and solves the key point drift problem, the average precision (mAP50) of the model in the classification task and key point detection task is effectively improved.

Description

Technical Field

[0001] This application relates to the field of image data processing, specifically a method for extracting key points from bird flight videos by integrating YOLO and LSTM networks. This method combines deep learning neural networks with time series modeling, accurately identifying target key points and correcting them using temporal information. It can be applied to scenarios such as bird behavior analysis, ecological monitoring, and biokinematics research. Background Technology

[0002] In avian functional morphology research, extracting key skeletal points has irreplaceable scientific value. Three-dimensional reconstruction based on these key skeletal points can analyze the mechanical transmission paths of critical components such as wing bones and phalanges, elucidating wing variation strategies employed by birds under different flight conditions.

[0003] With the iterative upgrades of computer vision algorithms based on deep learning, new methods for skeletal point analysis have emerged. Current technologies only require conventional camera equipment to acquire spatiotemporal images, and combined with feature point spatiotemporal correlation modeling algorithms, to output key motion indicators such as the target's displacement vector, velocity vector, and attitude angle. However, existing technologies have significant limitations, specifically as follows:

[0004] (1) Traditional methods rely on the static features of a single frame image, lack temporal information correlation, and are difficult to fully capture its dynamic change process, resulting in insufficient accuracy of key point analysis;

[0005] (2) When occlusion occurs, key points are easily missed or falsely detected, which seriously affects the continuity of the trajectory;

[0006] (3) When traditional deep learning models are used to handle specific tasks involving birds, if they are directly trained using random initialization or a single dataset, there is a risk of overfitting, which makes it difficult to meet the needs of complex scenarios in actual ecological monitoring.

[0007] How to improve the extraction of key points from bird flight videos has become an urgent technical problem to be solved. Summary of the Invention

[0008] The purpose of this invention is to address the problem that existing methods for extracting key points from bird skeletons using YOLO network technology rely on the static features of a single frame image, lack temporal information correlation, and are difficult to fully capture the dynamic changes, resulting in insufficient accuracy of key point analysis. This invention provides a method for extracting key points from bird flight videos that integrates YOLO and LSTM networks.

[0009] To achieve the above objectives, this application adopts the following technical solution.

[0010] A method for extracting key points from bird flight videos by integrating YOLO and LSTM networks includes the following steps:

[0011] Step 1: Based on bird data from open-source databases, collect and build a multi-perspective bird dataset;

[0012] Step 2: Perform dataset augmentation on the multi-view bird dataset obtained in Step 1 to obtain the augmented bird dataset;

[0013] Step 3: Input the multi-view bird dataset from Step 1 and the augmented bird dataset from Step 2 into the YOLOv8 network for training. By experimenting with the effects of different batch sizes and different optimization algorithms on the model results, select the optimal batch size and optimization algorithm, and obtain the pre-trained model.

[0014] Step 4: Concatenate the partial convolutional computation module (Partical_conv), the MLP layer composed of cascaded single-dimensional and two-dimensional convolutions, the dropout layer, and the EMA attention mechanism layer to form the CME module; based on YOLOv8, replace the Bottleneck sub-module in the C2f module with the CME module. The shortcut selection of the CME module is consistent with the shortcut selection of the Bottleneck sub-module in the original YOLOv8 network to construct the post-trained network; feed the pre-trained model and the skeletal keypoint data of the bird to be analyzed into the post-trained network for retraining to obtain the second post-trained model; the specific operations are as follows:

[0015] Step 4.1: Taking the birds to be analyzed as the object, collect images of this type of bird and the mesh image of the bird, perform target detection boxes and skeletal key point annotation, and obtain the post-training dataset;

[0016] Step 4.2: Convert the annotated post-training dataset from Step 4.1 into a skeletal keypoint dataset in YOLO network training format;

[0017] Step 4.3: Concatenate a partial convolutional computation module, an MLP layer (concatenated single-dimensional convolution and two-dimensional convolution), a dropout layer, and an EMA attention mechanism layer to form a CME module, replacing the Bottleneck submodule in the C2f module of YOLOv8. The rest of the network architecture remains unchanged. The shortcut selection of the CME module is consistent with the shortcut selection of the Bottleneck submodule in the original YOLOv8 network. The resulting network is then trained.

[0018] Step 4.4: Input the YOLO format skeletal keypoint data obtained in Step 4.2 into the post-trained network model in Step 4.3. The hyperparameters of the post-training are kept consistent with those of the pre-training. The training strategy has been modified as follows: First, freeze the parameters of the network model except for the C2f module and the detection head and train for 20-40 rounds. Then, unfreeze and train for another 50-60 rounds to obtain the second post-trained model.

[0019] Step 5: Use the second post-trained model obtained in Step 4 to predict the continuous flight video of the bird to be analyzed, and obtain the skeletal key point coordinate data for each frame.

[0020] Step 6: Perform time-series modeling correction to address target loss due to occlusion during flight, and complete the identification of key skeletal points of the bird to be analyzed. The specific operations are as follows:

[0021] Step 6.1: Connect the corresponding skeletal keypoints in the skeletal keypoint coordinate data obtained in Step 5 for every two frames to obtain the velocity vector and form a time-series dataset that reflects the dynamic evolution of the skeletal keypoints.

[0022] Step 6.2: Input the time series dataset obtained in Step 6.1 into the LSTM model for training to obtain the third time series prediction model;

[0023] Step 6.3: Use the second post-trained model to predict the flight video of the bird to be analyzed; when the target is occluded in the flight video, the confidence of the identified bird to be analyzed drops sharply or the key point shifts significantly, then the prediction result of the third time series prediction model in step 6.2 is used to replace the detection result of the second post-trained model.

[0024] In step 1, a general multi-view bird dataset is collected using open-source deep learning datasets available online.

[0025] In step 2, the images in the multi-view bird dataset obtained in step 1 are subjected to stable diffusion with random lighting perturbation and background replacement, and traditional data augmentation is performed using the OpenCV library to obtain an augmented bird dataset. The traditional data augmentation operations using the OpenCV library include the following: random rotation, random cropping, color dithering, and flipping.

[0026] In step 4.1, COCO-Annotator software is used to annotate the target detection bounding box and skeletal key points.

[0027] In step 4.1, there are 11 key skeletal points, including the head, chest, tail, left shoulder, right shoulder, left elbow, right elbow, left wrist point, right wrist point, left wingtip, and right wingtip.

[0028] In step 4.1, the target bird mesh image is obtained through the target bird 3D mesh, and bird data with different shapes, postures, sizes and perspectives can be generated by adjusting parameters.

[0029] In step 6.1, the corresponding skeletal key points in the skeletal key point coordinate data obtained in step 5 are connected in every two frames to form a velocity vector, and stored as a CSV file for LSTM model training.

[0030] In step 6.3, when the target is obstructed in the flight video, the confidence level of the identified bird to be analyzed drops sharply, or when the Euclidean distance deviation between the current frame detection value and the previous frame detection value exceeds a set percentage of the wingspan, the prediction result of the third temporal prediction model in step 6.2 is used instead of the detection result of the second post-trained model. The set percentage is 12%-18%. Preferably, the set percentage is 15%.

[0031] To address the aforementioned problems, this application provides a method for extracting key points from bird flight videos by integrating YOLO and LSTM networks. The bird skeletal point extraction method of this application, which combines deep learning and time-series modeling, can solve the problems of low accuracy, weak temporal consistency, poor robustness, and target loss due to occlusion in existing technologies, even with only a small number of labeled skeletal key points. As a high-precision, automated, and robust skeletal extraction method, this application has significant research value and practical application value for bird skeletal point extraction.

[0032] The relevant procedures for this application are as follows.

[0033] Step 1: Construct a multi-view bird dataset. Collect a large-scale, general-purpose multi-view bird dataset using open-source deep learning datasets available online.

[0034] In a specific instance, the large-scale general multi-view bird dataset in step 1 should collect multi-view bird datasets such as Birds-400 and CUB-200, covering more than 400 bird species.

[0035] Step 2: Perform dataset augmentation to obtain the augmented bird dataset. To further expand the dataset size and increase the model's generalization ability, various data augmentation techniques are employed, specifically: adding random lighting perturbations, background replacement, random rotation, random cropping, color dithering, and flipping. In Step 2, adding random lighting perturbations simulates bird images under different lighting conditions in real-world scenes; background replacement prevents the model from learning false features related to specific backgrounds, improving the model's detection ability in complex backgrounds; random rotation and flipping display bird morphology from different angles, enhancing the model's adaptability to changes in bird posture. For background replacement and random lighting perturbations, Stable-Diffusion technology was used: after deploying a local Stable-Diffusion Web user interface, images were input into Stable-Diffusion Web, corresponding masks were drawn, and then the required augmented images were generated. For flipping, random rotation, random cropping, and color dithering, traditional data augmentation methods were implemented using the OpenCV 4.5 library.

[0036] Step 3: Pre-training the Model: In the pre-training stage, the YOLOv8 network was selected as the basic pre-training network. The impact of different batch sizes and optimization algorithms on the model results was experimented with, and the optimal batch size and optimization algorithm were selected as the final pre-training model scheme. The collected open-source dataset, bird mesh images, and data-augmented images were input into the network for training to obtain the pre-trained model. In a specific example, three batch sizes (8, 16, and 32) and three optimization algorithms (SGD, Adam, and Adamx) were tried in Step 3, and the one with the best results was selected as the pre-training model.

[0037] Step 4, Second Post-Training Model: The post-training part is the process of retraining the pre-trained model for the keypoint detection task of birds to be analyzed. The specific steps are as follows.

[0038] Step 4.1, Dataset Collection: After identifying the bird species to be analyzed, images of the bird are collected, and meshed images are obtained using a 3D mesh model of the bird to obtain the post-training dataset. The bird mesh model can generate bird data with different shapes, postures, sizes, and perspectives by adjusting parameters, which can effectively compensate for the lack of bird data in the open-source dataset. Skeletal Keypoint Annotation: Object detection boxes and skeletal keypoints in a small number of acquired images are annotated using open-source annotation software to obtain the post-training dataset. In a specific example, the object box and keypoint annotation is done based on the open-source COCO-Annotator software. In a specific example, there are a total of 11 bird keypoints, including head, chest, tail, left shoulder, right shoulder, left elbow, right elbow, left wrist point, right wrist point, left wingtip, and right wingtip.

[0039] Step 4.2, Data Conversion: Convert the annotated post-training dataset from Step 4.1 into a skeletal keypoint dataset suitable for YOLO network training format.

[0040] Step 4.3, Post-trained Network: Since direct post-training using the YOLOv8 model results in significant skeletal keypoint bias, the network architecture needs improvement. Therefore, a CME module was designed and integrated into the YOLOv8 network to optimize the neural network structure, resulting in the post-trained network model. Some convolutional modules in Step 4.3 are based on a lightweight design philosophy. Through kernel optimization and reorganization, the computational load and parameter count are significantly reduced while retaining feature extraction capabilities, accelerating the network's forward propagation process. The EMA attention mechanism simulates the information focusing characteristics of the human visual system, dynamically allocating weights based on the correlation between image regions and bird targets, making the model more focused on bird information while ignoring background information.

[0041] Step 4.4, Second Post-Trained Model: Input the YOLO format skeletal keypoint dataset obtained in Step 4.2 into the post-trained network model in Step 4.3. The training hyperparameters (such as batch size, optimization algorithm, etc.) are consistent with the pre-trained model. The training strategy is to freeze the unmodified weights for 20-40 rounds and then unfreeze them for 50-60 rounds of overall training to obtain the second post-trained model.

[0042] Step 5: Model prediction and acquisition of skeletal keypoint data: The second post-trained model obtained in Step 4 is used to predict the continuous flight video of the bird to be analyzed and to obtain the skeletal keypoint coordinate data for each frame.

[0043] Step 6, Time Series Modeling Correction: Since the prediction results still have the problem of target loss due to local occlusion or multi-target occlusion during flight, a Long Short-Term Memory (LSTM) network model is introduced to solve this problem and complete the identification of the skeletal key points of the bird to be analyzed. The relevant operations are as follows.

[0044] Step 6.1: Constructing a Temporal Dataset: Since the position and pose changes of the same bird in adjacent frames of the video are relatively small, we can connect each corresponding keypoint in adjacent frames based on the obtained skeletal keypoint coordinates to form a velocity vector, thus creating a temporal dataset reflecting the dynamic evolution of the skeletal keypoints. In a specific example, the temporal dataset construction in Step 6.1 involves constructing velocity vectors from the stored key skeletal point coordinates and storing them as a CSV file for LSTM model training.

[0045] Step 6.2: Input the time-series dataset obtained in Step 6.1 into the LSTM model for training to obtain the third time-series prediction model. In a specific example, in Step 6.2, the velocity vector of the 11th frame is predicted using the velocity vectors of 10 consecutive frames. During this process, the LSTM model automatically learns the motion inertia law of key points changing over time.

[0046] Step 6.3, Prediction result correction: The second post-trained model is used to predict the flight video of the bird to be analyzed; when the target is occluded in the flight video, the confidence of the identified bird to be analyzed drops sharply or the key point is significantly shifted, then automatic calibration is performed, and the prediction result of the third time series prediction model in step 6.2 is used to replace the detection result of the second post-trained model.

[0047] In a specific instance, when the confidence of the detected bounding box drops sharply, or when the Euclidean distance deviation between the current frame detection value and the previous frame detection value exceeds 15% of the wingspan, the prediction value of the third temporal prediction model is used to replace the abnormal detection result.

[0048] Compared with existing technologies, this application effectively improves the model's generalization ability and solves the keypoint drift problem through a two-stage training strategy and network structure optimization. The average accuracy (mAP50) of the model is significantly improved in both classification and keypoint detection tasks. Simultaneously, this application effectively reduces joint tracking errors through an LSTM dynamic compensation mechanism, ensuring the temporal integrity of wing kinematic parameters. Using this application, the target loss problem in bird encounter occlusion scenarios was successfully solved in real-world testing. Attached Figure Description

[0049] The present invention will be described by way of example and with reference to the accompanying drawings, wherein:

[0050] Figure 1 This is an overall flowchart of the method for extracting key points from bird flight videos that integrates an improved YOLO network and time series modeling, which is the subject of this invention.

[0051] Figure 2 This refers to the data augmentation process based on Stable-Diffusion involved in the embodiment;

[0052] Figure 3 These are typical images from the dataset used by the pre-trained model involved in the embodiment;

[0053] Figure 4 This is a generalization test graph of the pre-trained model involved in the embodiment;

[0054] Figure 5 This refers to the gridded image involved in the embodiment;

[0055] Figure 6This is a schematic diagram of bird skeletal point extraction involved in the embodiment;

[0056] Figure 7 This is a schematic diagram illustrating the annotation of key skeletal points in the embodiment;

[0057] Figure 8 This is a schematic diagram of the post-trained network model improvement CME module involved in the embodiment;

[0058] Figure 9 This is a schematic diagram illustrating the integration of the post-trained network model into the convolution calculation module in the embodiment.

[0059] Figure 10 This is a schematic diagram illustrating the prediction effect of the post-trained model involved in the embodiment;

[0060] Figure 11 This is a roadmap for implementing LSTM model corrections in the embodiments;

[0061] Figure 12 This is a schematic diagram illustrating the principle of velocity vector construction in the embodiment.

[0062] Figure 13 The image shows the effect of the LSTM model correction involved in the embodiment. Detailed Implementation

[0063] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0064] Any feature disclosed in this specification, unless otherwise stated, may be replaced by other equivalent or similar features. That is, unless otherwise stated, each feature is merely one example of a series of equivalent or similar features. Example

[0065] This application uses birds as the final analysis object. The analysis object can be selected according to actual needs, such as eagles, sparrows, parrots, etc. In this embodiment, eagles are used as the final analysis object, and corresponding explanations are provided. The overall process is as follows: Figure 1 As shown.

[0066] This embodiment provides a method for extracting key points from bird flight videos by integrating YOLO and LSTM networks, which includes the following steps.

[0067] Step 1: Construct a multi-view bird dataset

[0068] Dataset Collection: Open source datasets such as Birds-400 and CUB-200 have been collected, covering more than 400 bird species.

[0069] Step 2: Obtain the augmented bird dataset

[0070] The multi-view bird dataset obtained in step 1 is augmented to obtain an augmented bird dataset. The specific steps are as follows: Random lighting perturbations and background replacements are added to the images in the multi-view bird dataset obtained in step 1 using Stable-Diffusion, as shown below. Figure 2 As shown, traditional data augmentation was performed using the OpenCV library to obtain an augmented bird dataset. The traditional data augmentation operations using the OpenCV library are as follows: random rotation, random cropping, color jittering, and flipping. A typical image of the final augmented bird dataset is shown below. Figure 3 As shown.

[0071] Step 3: Obtain the pre-trained model

[0072] The multi-view bird dataset from step 1 and the augmented bird dataset from step 2 were used as inputs to train the YOLOv8 network. Different batch sizes and optimizers were selected for training, and the best-performing model was chosen as the pre-trained model. Experimental results are shown in Tables 1 and 2, where Box(p) and R represent the precision (P) and recall (R) of the predicted bounding boxes, respectively; mAP50 represents the mean precision at an intersection-union (IU) threshold of 0.5; and mAP50-95 represents the mean precision from IU thresholds of 0.5 to 0.95, more rigorously measuring detection stability under different overlap requirements. By comparing various data, the best-performing optimization algorithm SGD and a batch size of 16 were selected. The generalization test results of the pre-trained model are shown below. Figure 4 As shown.

[0073] Table 1. Experimental results with different batch sizes when SGD is the fixed optimization algorithm.

[0074]

[0075] Table 2. Experimental results under different optimization algorithms with a fixed batch size of 16.

[0076]

[0077] Step 4: Based on the pre-trained model obtained in Step 3, retrain the model by targeting the key skeletal points of the eagle to obtain the second post-trained model. The specific operation is as follows.

[0078] Step 4.1 Collected images of eagles at different scales and generated eagle mesh images of different shapes, postures, and sizes based on the eagle mesh model. The mesh images are as follows: Figure 5 As shown, the key points of the skeleton are annotated in the annotation software, and the annotated model is as follows. Figure 6 As shown, typical annotation results are as follows: Figure 7 As shown in the figure. In a specific example, the annotation software used is COCO-Annotator.

[0079] Step 4.2: Convert the key point JSON format data obtained in Step 4.1 into a TXT file format suitable for the YOLO network.

[0080] Step 4.3: Concatenate a portion of the convolutional computation module, the MLP layer (concatenated single-dimensional and two-dimensional convolutions), the dropout layer, and the EMA attention mechanism to obtain the CME module. Replace the Bottleneck sub-module of the C2f module in the YOLOv8 network with the CME module, and then train the network model. A schematic diagram of the CME module is shown below. Figure 8 As shown, the specific architecture of some convolution calculation modules is as follows: Figure 9 As shown.

[0081] Step 4.4: Input the YOLO format skeletal keypoint dataset obtained in Step 4.2 into the post-trained network model in Step 4.3. The training hyperparameters (such as batch size, optimization algorithm, etc.) are consistent with the pre-trained model. The optimization algorithm is SGD, and the batch size is 16. The training strategy is to train for 30 epochs without modifying the network weights, and then unfreeze and train the whole model for another 50 epochs to obtain the second post-trained model.

[0082] Step 5: Using the second post-trained model obtained in Step 4, predict the continuous flight video of the eagle and obtain the skeletal keypoint coordinate data for each frame. The prediction results are as follows: Figure 10 As shown.

[0083] Step 6: Perform time-series modeling correction to address target loss due to occlusion during flight. This involves time-series-based correction of key skeletal points in birds, as detailed in the specific implementation process. Figure 11 As shown, the operation is as follows.

[0084] Step 6.1: Connect the corresponding bone keypoints in the bone keypoint coordinate data obtained in Step 5 for every two frames to form a velocity vector and generate the movement trajectory of each joint point over time, as shown in the diagram. Figure 12 As shown, a time-series dataset in SCV format is obtained, which reflects the dynamic evolution of skeletal keypoints.

[0085] Step 6.2: Input the time series dataset obtained in Step 6.1 into the LSTM model for training to obtain the third time series prediction model.

[0086] Step 6.3: Correct the prediction results using the third temporal prediction model obtained in Step 6.2. Specifically, use the second post-trained model to predict the flight video of the bird to be analyzed. When the target is occluded in the flight video, and the confidence of the identified bird to be analyzed drops sharply or the key points shift significantly, the prediction result of the third temporal prediction model in Step 6.2 is used instead of the detection result of the second post-trained model. In a specific example, compare the Euclidean distance deviation between the detection value of the current frame and the prediction value of the previous frame. If it exceeds 15% of the wingspan, the predicted value is used to replace the abnormal detection result. The correction result is compared to... Figure 13 As shown. Figure 13 (a) is the result without timing correction. Figure 13 (c) is Figure 13 (a) shows the result of time-series correction; Figure 13 (b) is the result without timing correction. Figure 13 (d) is Figure 13 (b) shows the results with time-series correction. The results show that the dynamic compensation mechanism based on the LSTM network effectively ensures the continuity of the key point time sequence and solves the problem of key point loss caused by occlusion.

[0087] To further verify the effectiveness of the proposed method, we conducted quantitative tests on whether a two-stage training strategy was adopted and whether modifications were made to the post-training network. The test results are shown in Tables 3 and 4. The analytical metrics used in the tables, from left to right, are: bounding box recall, average precision of bounding boxes with an intersection-union (IU) threshold of 50, average precision of bounding boxes with IU thresholds ranging from 50 to 95, keypoint precision, keypoint recall, average precision of keypoints with an IU threshold of 50, and average precision of keypoints with IU thresholds ranging from 50 to 95.

[0088] Table 3 compares the results of using a pre-training strategy. The YOLOv8 model using pre-training outperforms the model without pre-training in all dimensions of object bounding box detection and keypoint detection. This result can be explained from the perspective of feature transfer: the general features learned by pre-training can be effectively transferred to downstream detection tasks, which not only improves the recall rate of objects and keypoints, but also shows advantages in precision (mAP50-95) under strict intersection and union thresholds and fine localization of keypoints (Pose mAP50-95), reflecting the dual enhancement of model generalization ability and fine feature extraction ability by pre-training.

[0089] Table 4 presents the performance differences between YOLOv8 and the improved model YOLOv8-CME in the post-training stage. YOLOv8-CME achieves significant improvements in recall (R), mAP50, and mAP50-95 in the object bounding box detection dimension. This indicates that the CME improvement strategy effectively enhances the model's localization accuracy and recall capability for object bounding boxes, especially the improved accuracy stability (mAP50-95) under different intersection-union thresholds, reflecting the enhanced adaptability of the model to the diversity of object shapes. In the keypoint detection dimension (Pose), its recall rate and the mean accuracy under different accuracy thresholds also show significant gains, indicating that this improvement has a synergistic optimization effect on keypoint detection recall and fine localization performance, verifying the structural rationality of the CME strategy in multi-task detection scenarios.

[0090] Test results show that the two-stage optimization strategy effectively improves the model's generalization ability and successfully solves the skeleton keypoint drift problem by training the modified network. Furthermore, the dynamic compensation mechanism based on the LSTM network effectively reduces key tracking errors and ensures the temporal integrity of the wing kinematic parameters.

[0091] Table 3 Results of whether a pre-training strategy was used (all models are YOLOv8)

[0092]

[0093] Table 4 Results before and after model modification during training

[0094]

[0095] In summary, this application effectively improves the model's generalization ability and solves the problem of drift in extracted skeletal keypoints through a two-stage training strategy and post-training network structure optimization. Simultaneously, this application effectively reduces joint tracking errors through an LSTM dynamic compensation mechanism, ensuring the temporal integrity of wing kinematic parameters, and successfully solves the target loss problem in multi-target encounter and occlusion scenarios in actual tests.

[0096] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. However, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting key points from bird flight videos by integrating YOLO and LSTM networks, characterized in that, Includes the following steps: Step 1: Construct a multi-view bird dataset; obtain diverse data from this multi-view bird dataset; Step 2: Perform dataset augmentation on the multi-view bird dataset obtained in Step 1 to obtain the augmented bird dataset; Step 3: Input the multi-view bird dataset from Step 1 and the augmented bird dataset from Step 2 into the YOLOv8 network for training. By experimenting with the effects of different batch sizes and different optimization algorithms on the model results, select the optimal batch size and optimization algorithm, and obtain the pre-trained model. Step 4: Based on the pre-trained model obtained in Step 3, retrain the model for the key skeletal points of the bird to be analyzed to obtain the second post-trained model. The specific operation is as follows: Step 4.1: Taking the birds to be analyzed as the object, collect images of this type of bird, and select some of the images to perform object detection boxes and skeletal key point annotation to obtain the post-training dataset; Step 4.2: Convert the annotated post-training dataset from Step 4.1 into a skeletal keypoint dataset suitable for YOLO network training format; Step 4.3: Integrate the lightweight convolutional computation module and the EMA attention mechanism into the C2f module of the YOLOv8 network to obtain the post-trained network model; Step 4.4: Input the YOLO format skeletal keypoint dataset obtained in Step 4.2 into the post-trained network model in Step 4.

3. The training parameters are the same as those of the pre-trained model, and the second post-trained model is obtained. Step 5: Use the second post-trained model obtained in Step 4 to predict the continuous flight video of the bird to be analyzed, and obtain the skeletal key point coordinate data for each frame. Step 6: Perform time-series modeling correction to address target loss due to occlusion during flight, and complete the identification of key skeletal points of the bird to be analyzed. The specific operations are as follows: Step 6.1: Connect the corresponding skeletal keypoints in the skeletal keypoint coordinate data obtained in Step 5 for every two frames to form a velocity vector, thus forming a time-series dataset that reflects the dynamic evolution of skeletal keypoints. Step 6.2: Input the time series dataset obtained in Step 6.1 into the LSTM model for training to obtain the third time series prediction model; Step 6.3: Use the second post-trained model to predict the flight video of the bird to be analyzed; when the target is occluded in the flight video, the confidence of the identified bird to be analyzed drops sharply or the key point shifts significantly, then the prediction result of the third time series prediction model in step 6.2 is used to replace the detection result of the second post-trained model.

2. The method according to claim 1, characterized in that, In step 1, a large-scale, general multi-view bird dataset is collected using open-source deep learning datasets available online. To avoid the risk of overfitting in the detected birds, parameterized grid images of the birds to be analyzed are introduced to obtain diverse data.

3. The method according to claim 1, characterized in that, In step 2, the images in the multi-view bird dataset obtained in step 1 are subjected to random lighting perturbation and background replacement using Stable-Diffusion, and traditional data augmentation is performed using the OpenCV library to obtain an augmented bird dataset. The traditional data augmentation operations using the OpenCV library are as follows: random rotation, random cropping, color dithering, and flipping.

4. The method according to claim 1, characterized in that, In step 6.1, the corresponding skeletal key points in the skeletal key point coordinate data obtained in step 5 are connected in every two frames to form a velocity vector, and stored as a CSV file for LSTM model training.

5. The method according to claim 1, characterized in that, In step 6.3, when the target is obstructed in the flight video, the confidence level of the identified bird to be analyzed drops sharply, or when the Euclidean distance deviation between the current frame detection value and the prediction value of the third temporal prediction model exceeds the set percentage of the wingspan, the prediction result of the third temporal prediction model in step 6.2 is used to replace the detection result of the second post-trained model.

Images