A wild animal video target detection method based on a feature fusion network

Abstract

The application discloses a wild animal video target detection method based on a feature fusion network, and comprises the following steps: collecting and labeling wild animal image and video data; performing morphological transformation, color transformation and contour destruction preprocessing; sequentially inputting the preprocessed image and video data into a feature extraction network to obtain multi-scale features; performing feature fusion on the extracted multi-scale features, fusing multi-scale features and same-scale features of adjacent image frames for the video data, and only fusing multi-scale features for the image data, finally obtaining multi-scale fusion features; inputting the multi-scale fusion features into a feature prediction network to obtain a prediction result; comparing the prediction result with a label in the training stage, calculating a loss function, performing gradient back propagation, completing model parameter updating, and until the network converges; in the inference stage, the prediction result is post-processed to obtain the final output result.

Description

Technical Field

[0001] This invention relates to a video target detection method, and more particularly to a wildlife video target detection method based on feature fusion networks. Background Technology

[0002] The discovery and statistics of wildlife populations in the wild have always been a crucial issue in environmental protection. By analyzing data on the current distribution and activity status of wild animals, experts can gain insights into the current species diversity and ecological environment of protected areas, which is essential for further environmental protection efforts. Traditionally, the method of having environmental protection workers photograph wild animals in protected areas and then having experts identify and statistically analyze the targets requires significant manpower and resources, and is also time-consuming.

[0003] Video object detection algorithms perform object detection on each frame of a video to identify targets of interest, classify them, and determine their location. For example, they might detect the presence of a wild boar in an image and mark its location. However, due to blurring and unusual poses in video frames, well-developed static object detection methods often fail to achieve satisfactory results when directly applied to video object detection tasks.

[0004] Common video object detection algorithms mainly include tracking-based, feature propagation-based, and optical flow-based multi-frame aggregation methods. Tracking-based methods combine detection and tracking in the early stages, no longer aggregating the two sets of bounding boxes predicted by the detector and tracker together. Instead, they generate a separate set of bounding boxes based on the target detector's output and the trajectory calculated in previous frames. Feature propagation-based methods run deep convolutional neural networks only on sparse keyframes, then propagate the extracted features through another shallow neural network to other non-keyframes. Optical flow-based multi-frame aggregation methods calculate the extracted feature information for each frame, then calculate the transformation between the feature information of adjacent frames, and obtain the final result through a series of weighted sums. The current frame benefits from the previous frames.

[0005] All the video object detection algorithms mentioned above require labeled videos as training data. However, in practical applications, labeling video data is a huge undertaking and very costly. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for detecting wildlife video targets based on feature fusion networks, which addresses the shortcomings of the existing technology.

[0007] To address the aforementioned technical problems, this invention discloses a method for detecting wildlife targets in videos based on feature fusion networks, comprising the following steps:

[0008] Step 1: Collect wildlife video and image data. bird The top-left corner coordinates, length, and width of the smallest bounding box completely containing the target, along with the target's category information, are used to create a label. bird ;

[0009] Step 2, process the collected wildlife data. bird Data preprocessing is performed, including morphological transformation, color transformation, and contour destruction preprocessing on the acquired image data, and contour destruction preprocessing on the acquired video data, resulting in a preprocessed dataset. preprocess ;

[0010] Step 3, process the preprocessed dataset Data preprocess Input to the feature extraction network Model sele Feature extraction is performed to obtain features at multiple scales, i.e., multi-scale features (FM). select ;

[0011] Step 4: Input the multi-scale features into the feature fusion network Model. fuse Feature fusion is performed on the dataset Data. preprocess The video data in the dataset is fused with features of adjacent image frames at the same scale and features of the same image frame at different scales, and the fused features of each frame are saved; for the dataset Data preprocess The image data in the process only fuses features at different scales of the image, and finally obtains the multi-scale fusion feature FM. fuse ;

[0012] Step 5, fuse multi-scale features FM fuse Input to the prediction network Model predict In the process, the top-left corner coordinates and width and height values ​​of the predicted bounding boxes at multiple scales are obtained, as well as the category to which the target in the predicted bounding box belongs. The NMS algorithm is used to merge duplicate detection results to obtain the final predicted value Y. predict ;

[0013] Step 6: During the network training phase, calculate the prediction result Y using the loss function Loss. predict and the actual label bird The loss value between the prediction network Model predict Backpropagation begins, fixed feature fusion network Model fuse The parameters of the adjacent image frame feature fusion module are updated to update the feature fusion network Model. fuse The parameters of the multi-scale feature fusion module in the text;

[0014] Step 7: In the network inference stage, the network prediction result Y of the image data of the target to be detected is used. predict Directly used as the final output Y output For the prediction result Y of video data predict Based on the actual application scenario, post-processing filtering is performed to obtain Y. output .

[0015] Beneficial effects:

[0016] This invention proposes a feature fusion-based method for target detection in wild animal videos. By combining static target detection methods with video context and temporal information through a feature fusion module, the method achieves more accurate results when applied to video target detection. This method can be trained using both image and video data simultaneously, significantly expanding the training set and further improving detection accuracy. Furthermore, this invention also proposes a data augmentation method that can significantly reduce the false detection of background information. Attached Figure Description

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0018] Figure 1 This is a schematic diagram of the process of the present invention.

[0019] Figure 2 This is a schematic diagram of the contour destruction effect in this invention.

[0020] Figure 3 This is a schematic diagram of the feature fusion network in this invention. Detailed Implementation

[0021] This invention provides a feature fusion-based method for wildlife video target detection. Addressing the issues of limited video annotation data and the poor performance of static target detection methods on video data, this invention offers a feature fusion-based video target detection method that can be trained using both image and video annotation data, significantly expanding the training dataset. Furthermore, it can extend static target detection models by incorporating contextual and temporal information from the video, maintaining excellent detection accuracy and speed. To address the complex backgrounds in wildlife conditions where models focus more on target contour information, a data augmentation method is proposed to mitigate the interference of target contour information on the model. Figure 1 As shown, a wildlife video target detection method based on feature fusion includes the following steps:

[0022] Step 1: Collect wildlife video and image data. birdThen, using open-source annotation tools, the top-left corner coordinates, width, and height of the smallest bounding box completely containing the target, along with the target's category information, are used to create a label. bird ;

[0023] Step 2, process the collected wildlife data. bird Data preprocessing is performed, including morphological transformation, color transformation, and contour disruption preprocessing on the acquired image data. Morphological transformation includes randomly rotating targets in the image by 180° or flipping them horizontally and vertically, and color transformation. Contour disruption preprocessing is performed on the acquired video data to obtain the preprocessed dataset. preprocess ;

[0024] Step 3, process the preprocessed dataset Data preprocess Input to the feature extraction network Model select Feature extraction is performed to obtain features FM at multiple scales. select ;

[0025] Step 4: Input the multi-scale features into the feature fusion network Model. fuse For video data, features of the same scale from adjacent image frames and features of different scales from the same image frame are fused, and the fused features of each frame are saved. For image data, only features of different scales of that image are fused, and finally, a multi-scale fused feature FM is obtained. fuse ;

[0026] Step 5, fuse multi-scale features FM fuse Input into the prediction network Model predict The algorithm obtains the top-left corner coordinates and width and height values ​​of the predicted bounding boxes at multiple scales, as well as the category of the target within the predicted bounding box. Then, it uses the NMS algorithm (see: "Efficient non-maximum suppression." 18th International Conference on Pattern Recognition (ICPR'06). Vol.3. IEEE, 2006.) to merge duplicate detection results, yielding the final predicted value Y. predict ;

[0027] Step 6: During the network training phase, calculate the prediction result Y using the loss function Loss. predict and actual label bird The loss values ​​between adjacent image frames are backpropagated in the network to fix the parameters of the feature fusion module of adjacent image frames and update the parameters of the multi-scale feature fusion module.

[0028] Step 7: In the network inference stage, the network prediction result Y of the image is... predictDirectly used as the final output Y output For the prediction result Y of video data predict Based on the actual application scenario, post-processing filters out unreasonable prediction results to obtain Y. output .

[0029] A further improvement of the wildlife video target detection method of the present invention is as follows:

[0030] The contour destruction preprocessing method in step 2 includes the following steps:

[0031] Step 2-1: Set the initial random value r = random(0,1);

[0032] Step 2-2: When r < 0.5, output the original image directly without any modification;

[0033] Steps 2-3: When r ≥ 0.5, for the i-th image... The j-th detection target Its upper left corner coordinate is (x i,j ,y i,j ), with a length of w i,j Width is h i,j ;

[0034] Steps 2-4, according to the formula Calculate the occlusion width According to the formula Calculate the occlusion interval

[0035] Steps 2-5, from the target Starting from the top left pixel, from left to right, every... Each pixel will The pixel value of each pixel is set to 0 until it exceeds the target. The pixel range is then processed, and each row of pixels is processed from top to bottom.

[0036] Step 3 includes the following steps:

[0037] Step 3-1, Training Phase: First, input all wildlife image data, then input all video data;

[0038] Step 3-2, using the feature extraction network Model select We chose to use the Darknet53 network model (reference: "Yolov3:An incremental improvement." arXiv preprint arXiv:1804.02767(2018).) to extract features at three different scales;

[0039] Step 3-3, for image data Multi-scale features (FM) are extracted directly. select ;

[0040] Steps 3-4, for video data The video data is split into image frames, each image... The corresponding multi-scale feature FM will be extracted. select .

[0041] Step 4, the feature fusion method, includes the following steps:

[0042] Step 4-1, Feature Fusion Network Model fuse Includes the adjacent image frame feature fusion module Model fuse1 Model and multi-scale feature fusion module fuse2 Multi-scale features FM obtained from feature extraction select It will sequentially pass through the adjacent image frame feature fusion module Model fus Model and multi-scale feature fusion module fuse2 Finally, the multi-scale fusion feature FM is obtained. fuse ;

[0043] Step 4-2: Extract the features of scale k from the nth frame image. Input to the feature fusion module Model fuse1 According to the formula By fusing features of adjacent image frames at the same scale, for a single image and the first frame of a video, the parameters... The value is 1; for images that are not in the first frame of the video, the parameter... The value is 0.5;

[0044] Step 4-3, Multi-scale Feature Fusion Module Model fuse2 It contains two hidden layers, consisting of FPN (see: "Feature pyramid networks for object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.) and PANet (Mei, Yiqun, et al. "Pyramid attention networks for image restoration." arXiv preprint arXiv:2004.13824(2020.)). The first hidden layer will combine the features... Through calculation formula

[0045]

[0046] Get the output of the first hidden layer The second hidden layer will calculate thrd using the formula.

[0047]

[0048] Convert to

[0049] The post-processing method for video data in step 7 includes the following steps:

[0050] Step 7-1, the network's inference results include the predicted top-left corner coordinates of the target (x... predict ,y predict ), Length and width values ​​w predict and h predict Target category ID predict and confidence level conf predict The image has a length of w and a width of h;

[0051] Step 7-2, Filtering And conf predict Prediction result <0.95;

[0052] Step 7-3, Filtering And conf predict Prediction result <0.95;

[0053] Step 7-4, filter conf predict Prediction results <0.4;

[0054] Step 7-5, filter w predict ×h predict For prediction results less than 1024, the final output Y is obtained. output .

[0055] Example:

[0056] See Figure 1 The present invention provides a wildlife video target detection method based on feature fusion networks, comprising the following steps:

[0057] Step 1: Data Acquisition and Labeling. In this example, the wildlife data collected includes image and video data. Data sources include open-source, pre-labeled wildlife datasets, wildlife images and videos captured by cameras in the wild, and environmental background image data, forming the dataset. birdFor unlabeled data, manual annotation was performed using the open-source DarkLabel annotation tool. The annotations included the top-left corner coordinates of the smallest bounding box that completely contains the target, the box's length and width, and the target's category information, forming the corresponding true label for the data. bird .

[0058] Step 2, data preprocessing, involves data augmentation to expand the dataset and improve model robustness, including morphological transformation, color transformation, and contour destruction. In this example, this specifically involves morphological and color transformations and contour destruction on the images, contour destruction on the acquired video data frame by frame, and finally merging them into a complete video. The morphological transformation specifically includes rotating the target by 180°, flipping it horizontally, and flipping it vertically. The contour destruction specifically includes the following steps:

[0059] Step 2-1: Set the initial random value r = random(0,1)

[0060] Step 2-2: When r < 0.5, output the original image directly without any modification;

[0061] Steps 2-3: When r ≥ 0.5, for the i-th image... The j-th detection target Its upper left corner coordinate is (x i,j ,y i,j ), with a length of w i,j Width is h i,j ;

[0062] Steps 2-4, according to the formula Calculate the occlusion width According to the formula Calculate the occlusion interval

[0063] Steps 2-5, from the target Starting from the top left pixel, from left to right, every... Each pixel will The pixel value of each pixel is set to 0 until the target is exceeded. The pixel range is then processed, and each row of pixels is processed from top to bottom, resulting in the following effect: Figure 2 As shown.

[0064] Step 3, feature extraction. In this example, this involves randomly dividing the preprocessed data and background image data into a training set, validation set, and test set in a 3:1:1 ratio as positive and negative samples, which then serve as input to the feature extraction network. The training and validation sets are used to train the model, while the test set is used to evaluate its prediction performance. During training, in each round of inputting training data, all image data is randomly input first, followed by random input of video data, which is split into image frames and input sequentially. The feature extraction network chosen is Darknet53, which contains 53 convolutional layers, with a downsampling rate of [missing value]. The features at the three scales are used as the output of this layer.

[0065] Step 4, Feature fusion, the feature fusion network is as follows: Figure 3 As shown, in this example, it is specifically represented by the feature fusion network Model. fuse Includes the adjacent image frame feature fusion module Model fuse1 Model and multi-scale feature fusion module fuse2 Multi-scale features FM obtained from feature extraction select It will sequentially pass through the adjacent image frame feature fusion module Model fuse1 Model and multi-scale feature fusion module fuse2 Finally, the multi-scale fusion feature FM is obtained. fuse Features of scale i Input to the feature fusion module Model fuse1 According to the formula Features of adjacent image frames at the same scale are fused. For a single image and the first frame of a video, the parameters... The value is 1, for other frame parameters of the video. The value is 0.5. Multi-scale feature fusion module (Model) fuse It contains two hidden layers, consisting of an FPN and a PAN stack. The first hidden layer will store the features. Through calculation formula

[0066] Get the output of the first hidden layer The second hidden layer will... Through calculation formula

[0067]

[0068] Convert to

[0069] Step 5, Feature Prediction: Analyze the multi-scale fusion features obtained from the feature fusion network to predict the location and category information of the target in the input data. In this example, this is specifically achieved by using the YoloHead network (reference: "Yolov4: Optimal speed and accuracy of object detection." arXiv preprint arXiv:2004.10934(2020).) to perform convolution operations on the multi-scale fusion features, obtaining prediction results at three scales, including the predicted top-left corner coordinates and width and height values ​​of the target, as well as the target's category information. The network prediction results are then converted into coordinate values ​​of the original image. Finally, the NMS algorithm is used to merge duplicate prediction results. Specifically, this involves calculating the overlap between any two prediction results of the same category. Where Area represents the coordinates of the prediction box. When the overlap is greater than 0.5, predictions with lower confidence will be deleted.

[0070] Step 6, Model Training: During model training, the loss value between the predicted results and the true labels is calculated, and then backpropagated throughout the network to adjust the model parameters to achieve the best prediction results. In this example, the loss function is set to Loss = DIOULoss + BCELoss (refer to "Yolov4: Optimal speed and accuracy of object detection." arXiv preprint arXiv:2004.10934(2020).), the learning rate is set to 0.001, the batch size is 64, the weight decay is 0.0005, and the training device is two Nvidia 2080ti graphics cards.

[0071] Step 7, Model Inference: During the network's inference phase, filtering rules are set according to the actual application scenario to ensure the effectiveness of the inference. In this example, this is specifically manifested in the results including the predicted top-left corner coordinates of the target (x, y). predict ,y predict ), Length and width values ​​w predict and h predict Target category ID predict and confidence level conf predict The image has a length of w and a width of h;

[0072] Step 7-2, Filtering And conf predict Prediction result <0.95;

[0073] Step 7-3, Filtering And conf predict Prediction result <0.95;

[0074] Step 7-4, filter conf predict Prediction results <0.4;

[0075] Step 7-5, filter w predict ×h predict For prediction results less than 1024, the final output Y is obtained. output .

[0076] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a wildlife video target detection method based on a feature fusion network, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0077] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0078] This invention provides an idea and method for wildlife video target detection based on feature fusion networks. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A wild animal video target detection method based on a feature fusion network, characterized in that, The method comprises the following steps: Step 1, collect wild animal video and image data And label the left upper corner coordinate value and length-width value of the minimum rectangle frame containing the target completely, and the category information of the target, to form a label ; Step 2, data preprocessing is performed on the collected wild animal data The data preprocessing is performed, wherein morphological transformation, color transformation and contour destruction preprocessing are performed on the collected image data, and contour destruction preprocessing is performed on the collected video data to obtain a preprocessed data set ; Step 3, the preprocessed data set is input into a feature extraction network to perform feature extraction to obtain features of multiple scales, i.e., multi-scale features to perform feature extraction to obtain features of multiple scales, i.e., multi-scale features ; Step 4, inputting the multi-scale features into a feature fusion network to perform feature fusion ; For video data in dataset , the same scale features of adjacent image frames and different scale features of the same image frame are fused, and the fused features of each frame are saved; for image data in dataset , only different scale features of the image are fused, and finally multi-scale fused features are obtained. Step 5, the multi-scale fusion feature Input to the prediction network , get the prediction box left upper corner coordinates and length-width values of multiple scales, and the category of the target in the prediction box, use NMS algorithm to merge repeated detection results, and get the final prediction value ; Step 6, in the network training stage, using the loss function Calculate the loss value between the predicted result And the actual label From the prediction network Start back propagation, fix the parameters of the adjacent image frame feature fusion module in the feature fusion network Update the parameters of the multi-scale feature fusion module in the feature fusion network ​ Step 7, in the network inference stage, the network prediction result of the image data of the target to be detected directly as the final output , for the prediction result of the video data , according to the actual application scene, post-processing filtering is carried out to obtain ; The profile destruction preprocessing in step 2 comprises the following steps: Step 2-1, setting random initial value ; Step 2-2, when The original drawing is output directly without any modification. Step 2-3, when , for the jth detection target in the ith image , whose upper-left corner coordinate is , length is , and width is , the following is calculated: ​ Step 2-4, calculate the occlusion width The method is as follows: ； Computing occlusion intervals The method is as follows: ； Steps 2-5, from the detection target Starting from the top left pixel, from left to right, every... Each pixel will The pixel value of each pixel is set to 0 until it exceeds the detection target. The pixel range; processing the rows of pixels of the detection target from top to bottom according to the method . 2.The wild animal video target detection method based on the feature fusion network according to claim 1, wherein, The feature extraction in step 3 comprises the following steps: Step 3-1, training phase, in the feature extraction network the wild animal video and image data in the image data, and then input the video data; Step 3-2, feature extraction network In the present embodiment, a Darknet53 network model is selected to extract three features of different scales. Step 3-3, for image data , directly extract multi-scale features ; Step 3-4, for video data Splitting the video data into image frames, each image frame Extracting corresponding multi-scale features . 3.The wild animal video target detection method based on the feature fusion network according to claim 2, characterized in that, The feature fusion network described in step 4 , comprising: an adjacent image frame feature fusion module and a multi-scale feature fusion module . 4.The wild animal video target detection method based on the feature fusion network according to claim 3, characterized in that, The feature fusion in step 4 comprises the following steps: The multi-scale features extracted in step 3 are input into a multi-scale feature fusion module The adjacent image frame feature fusion module And the multi-scale feature fusion module Finally, the multi-scale fusion features are obtained . 5.The wild animal video target detection method based on the feature fusion network according to claim 4, characterized in that, The feature fusion in step 4 comprises the following steps: Step 4-1, using the adjacent image frame feature fusion module fusing the features of the same scale of the adjacent image frames; Step 4-2, multi-scale feature fusion module is used to perform multi-scale feature fusion. 6.The wild animal video target detection method based on the feature fusion network according to claim 5, wherein, The fusion of the features of the adjacent image frames in step 4-1 comprises the following steps: The first frame image extracts the scale feature is input to the feature fusion module , and the features of the same scale in adjacent image frames are fused, and the method is as follows: ； wherein, represents the first frame image obtained by the feature fusion module, represents the first frame image obtained by the feature fusion module, is a parameter.

7. The wild animal video target detection method based on the feature fusion network according to claim 6, characterized in that, In step 4-1, for single image and video first frame image, parameter is set as 1; for video other frame image, parameter is set as 0.

5. 8.The wild animal video target detection method based on the feature fusion network according to claim 7, characterized in that, The multi-scale feature fusion module described in step 4-2 Composed of two layers of hidden layers, composed of FPN and PAN superposition; The first layer hidden layer hides features The conversion is performed in the following manner: ； wherein, represents the output of the first layer hidden layer of scale k, represents the feature of scale k obtained by the adjacent frame feature fusion module, represents a convolution operation; represents a stacking operation; The second layer of hidden layers will Converting, the method comprising: ； wherein, denotes the feature of scale k obtained by the multi-scale feature fusion module, denotes the output of the first hidden layer of scale k+1. 9.The wild animal video target detection method based on the feature fusion network according to claim 8, characterized in that, The post-processing filtering in step 7 comprises the following steps: Step 7-1, the prediction result includes: predicted target upper left corner coordinates , length-width value and , target category and confidence , image length and width in the video data; Step 7-2, filtration and the prediction result of Step 7-3, filtration and the prediction result; Step 7-4, filtration the prediction results; Step 7-5, filtration the prediction results of the last output .

Images