A macaque detection method based on a YOLOv7 network and a Deepsort network

Abstract

The application discloses a macaque detection method based on a YOLOv7 network and a Deepsort network, inputs a macaque motion video sequence into the network, extracts and processes frame images, obtains a prediction result by extracting features of the input images through the optimized YOLOv7 network, sends the prediction result into the Deepsort network again as input information to obtain a prediction frame sequence, and finally detects and processes missing values of the prediction frame sequence. The application is based on a single-stage target detection algorithm, combines a time sequence transfer module (TSM) to extract space-time information of a target video sequence, improves the precision of macaque detection, concatenates a Deepsort algorithm to the detection output position to re-process and optimize the output result, and solves the problems of difficulty in feature selection, low recognition rate, influence of factors such as brightness, contrast change and shadow in the image, and low detection precision under the influence of cage shielding.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and computer vision, and particularly relates to a macaque detection method based on a YOLOv7 network and a Deepsort network. BACKGROUND

[0002] With the simplification and popularization of monitoring equipment, the number of monitoring videos is increasing day by day, especially in animal behavior experiments, it is difficult to obtain useful information from a large number of monitoring videos. In the past, only manual viewing of the video at high speed and recording were available, but this method obviously consumes manpower and time. Existing detectors can be roughly divided into two stages (for example, Faster-RCNN) and single stage (for example, YOLO). The two-stage detector first generates region proposals that may be targets on the image, and classifies and fine-tunes these proposals in the classification stage to fit the target contour. The single-stage detector does not give region proposals, but directly generates the class probability and position coordinates of the object on the original image, and the final result can be obtained through a single detection. Generally, the two-stage detector has higher accuracy but slower speed, and the single-stage detector has faster speed but lower accuracy. But with the update of detection technology, the single-stage detector not only has faster speed than the two-stage detector, but also has similar accuracy.

[0003] Object tracking is an important problem in the field of computer vision. In recent years, tracking by detection has received increasing attention. It is a follow-up task of object detection, taking video sequences as input, and obtaining one or more target prediction boxes and IDs through target positioning, feature extraction, data association, and trajectory updating. Deepsort is such a method.

[0004] Traditional 2D convolutional neural networks have fast computing speed, but cannot capture temporal relationships. Methods based on 3D convolutional neural networks can achieve good performance but have high computational complexity and deployment cost. SUMMARY

[0005] The purpose of the present application is to solve the problems existing in the prior art and provide a macaque detection method based on a YOLOv7 network and a Deepsort network. The present application is based on a single-stage object detection algorithm, combines a TSM time sequence transfer module to extract the spatiotemporal information of the target video sequence, improves the accuracy of macaque detection, and processes and optimizes the output results by concatenating the Deepsort algorithm to the target detection output position, thereby solving the problems of difficult feature selection, low recognition rate, influence of image brightness, contrast change and shadow, and low detection accuracy under the influence of cage shielding.

[0006] Technical scheme: A macaque detection method based on YOLOv7 network and Deepsort network, comprising the following steps:

[0007] Step S1, input the video sequence to the network

[0008] The video frames of the video are captured by using the opencv tool package and the frame images are returned, all frame images are respectively resized (for example, can be resized to 640x640) according to the order of the original video sequence, and the frame images after resizing are sequentially input to the YOLOv7 network;

[0009] As shown in Figure 3 , since the backbone of the existing YOLOv7 can only extract the information of a single image, even adjacent images will not have any intersection, but in video target detection, adjacent images input are usually very similar, so if the information is exchanged on adjacent images, exchanging information on adjacent images should obtain more robust video features, therefore, the present application starts from the P2 layer in the backbone layer of the YOLOv7 network, and each BConv layer includes a ShiftConv convolutional layer, a BN layer and an activation function;

[0010] Step S2, extracting features of the input frame image through the YOLOv7 network

[0011] For the frame image input to the backbone layer, the ShiftConv layer selects several frame images to exchange channel information according to the set batch size (≥2), then the head layer of the YOLOv7 network outputs three layers of feature maps of different sizes, and the output processed feature maps are obtained through convolution; finally, the prediction result is obtained by predicting the bounding box on the processed feature map;

[0012] This step retains the original structure design of the Head layer in YOLOv7, and the output three different scale feature maps are used for prediction, different size feature maps have different receptive fields, and fusion of three scales can process different scale target prediction; in the above feature extraction process, the ShiftConv layer exchanges information on adjacent images to extract the time sequence information of different frame images in the video sequence;

[0013] Step S3, establishing the macaque motion trajectory path

[0014] The prediction result is input into the Deepsort network again to assign an id to each target, and then a track path is established for each target bounding box bbox, followed by connection into a bbox sequence and removal of abnormal bbox away from the track path, and the Deepsort network finally outputs the target bbox prediction frame of each frame image in sequence, thereby obtaining a prediction frame sequence;

[0015] The application further emphasizes the time sequence relationship between consecutive frames by inputting the YOLOv7 output result into the deepsort network, and only uses the deepsort to establish a track path according to the spatial position and time sequence relationship without making any changes to the deepsort.

[0016] In step S4, the prediction frame sequence obtained in step S3 is detected and processed for missing values, which is based on the assumption that the position of a moving object is continuous in time and space, and is not limited to macaque detection, but can be extended to position or trajectory verification of other moving objects.

[0017] Further, the backbone layer in the YOLOv7 network in step S1 includes a plurality of BConv layers, E-ELAN layers and MPConv layers, the SiLU function is used as the activation function in the YOLOv7 network, and the E-ELAN layers and the MPConv layers are used as feature networks.

[0018] Further, the slice size of the ShiftConv convolution layer is 4, the exchange ratio is 1 / 4, and the original input dimension and output dimension are kept unchanged. For example, each output (x, y, w, h, o) contains coordinate positions and front and back backgrounds, the anchor quantity is 3, and the output of each layer is (1+5)x3 = 18, and the feature map size is multiplied by the final output of the detector; since the output of the general detector is obtained by only relying on a single image, this prediction only relies on a single image without considering the time sequence information of the video sequence, therefore, the output of the detector is input into the Deepsort network again; the Deepsort network matches and classifies the predicted bbox sequence, assigns a unique id to each target and establishes a respective track path, associates the prediction results of a single image, corrects the results of each frame to obtain the final output.

[0019] In the application, the animal target to be detected is assumed to appear in each frame image, if there is no prediction frame in the frame image, it is considered that there is a target in the frame image but the prediction result is not successfully output due to the detector, and then a linear interpolation method is used to fill in the prediction frame of the frame image; at the same time, according to the x direction and y direction coordinates and speed information of the prediction frame, it is judged whether there is a false detection, if there is a false detection, the false detection result is deleted and the value is filled in by the linear interpolation method.

[0020] Advantages: Compared with the prior art, the present application has the following advantages:

[0021] (1) The present application inserts a time sequence shift module TSM into a two-dimensional convolutional neural network, moves part of the channels along the time dimension, facilitates information exchange between adjacent frames, realizes time modeling with zero calculation and zero parameters, and realizes low delay of 13 milliseconds and 35 milliseconds for online video recognition.

[0022] (2) The present application combines the YOLOv7 network with the TSM time sequence shift module to extract the space-time information of the target video sequence, improves the precision of macaque video detection, processes and optimizes the output results in series by Deepsort algorithm, solves the problems of difficult feature selection and low recognition rate in existing video target detection, and the detection precision is not high under the influence of image brightness, contrast change and shadow, and cage shielding. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is the overall detection flowchart of the present application;

[0024] Figure 2 is a ShiftConv layer diagram in the present application;

[0025] Figure 3 is a diagram of the existing YOLOv7 network structure model;

[0026] Figure 4 is a diagram of the improved YOLOv7 model of the present application;

[0027] Figure 5 is a complete framework diagram of the present application;

[0028] Figure 6 is a feature map extracted by the network and a visualization result. DETAILED DESCRIPTION

[0029] The technical scheme of the present application will be described in detail below, but the protection scope of the present application is not limited to the described embodiments.

[0030] As shown in the figure, the macaque detection method based on the YOLOv7 network and the Deepsort network of the present application comprises the following steps: Figure 1

[0031] Step S1, input the video sequence into the network

[0032] ​The video frame of the video is captured by using the opencv toolkit and the frame image is returned, all frame images are respectively resized in the order of the original video sequence, and the resized frame images are sequentially input into the YOLO network; as shown in Figure 2 and Figure 4 As shown, starting from the P2 layer in the backbone layer of the YOLO v7 network, each BConv layer includes a ShiftConv convolutional layer, a BN layer and an activation function.

[0033] Unlike fusing the output results at the network level, the improved YOLO v7 network herein fuses the TSM module (ShiftConv convolutional layer) in the network structure, and updates the network structure to obtain high-quality spatiotemporal features.

[0034] Step S2, extracting features of the input image by YOLOv7

[0035] For the frame image input into the backbone layer, the ShiftConv layer selects several frame images to exchange channel information according to the set batch size, then the head layer of the YOLOv7 network outputs three layers of feature maps of different sizes, and the output processed feature maps are obtained through convolution; finally, the prediction result is obtained by predicting the bounding box on the processed feature map; in this embodiment, the batch size is set to 8, that is, the time information is exchanged on eight consecutive images, and the spatiotemporal features are extracted.

[0036] Step S3, establishing the motion trajectory path of the macaque

[0037] The prediction result is again input into the Deepsort network as input information to assign an id to each target, then a trajectory path is established for the bounding box bbox of each target, then a bbox sequence is connected and the abnormal bbox far from the trajectory path is removed, and the Deepsort network finally outputs the target bbox prediction frame of each frame image in order, and thus a prediction frame sequence is obtained.

[0038] Step S4, detecting and processing the missing values of the prediction frame sequence obtained in step S3.

[0039] As shown in Figure 5As shown, the present invention first selects the animal target motion video to be detected, and then, according to the prompts, selects the detection range (i.e., the area where the target may appear) on the pop-up window to crop the required length. If no selection is made, the entire image will be detected by default. After the selection is completed, the network will crop the image and resize the cropped image to the specified size and input it into the detector (the improved YOLOv7 network). After one stage of detection, the target category, predicted bounding box coordinates, and confidence score are directly output. The detection results from the detector are then input into the Deepsort network for trajectory association. The Deepsort network receives the output of the first-stage detector as input, assigns an ID to each target, and establishes a trajectory path between consecutive frames. If the predicted bounding box of the target with the corresponding ID deviates from the matching trajectory path, the predicted bounding box is discarded. Deepsort will give the estimated value of the current frame based on the prediction values ​​of past frames and output this value as the predicted bounding box of the current frame. This ensures that the prediction of each frame is on the matching trajectory path. The final output of the Deepsort network is the predicted bounding box of the target for each frame. The predicted bounding box sequence can be obtained by outputting the bounding boxes in the order of the video. Since the improved YOLOv7 network (detector) may still have false positives and false negatives, linear interpolation is used to fill in the missing values ​​if false positives and false negatives occur.

[0040] Example 1:

[0041] This embodiment uses target motion detection on macaques, mainly collecting video recordings from two locations and 13 monkeys over five days.

[0042] During testing, a 1-minute segment was extracted from each time period for each macaque to ensure the test set contained various interference factors such as lighting and occlusion. Considering the low movement frequency of macaques in their cages, video frames were output from the video slices at a frame rate of 5 FPS for annotation. Each image was labeled with only a bounding box and assigned a category ID. The final training set contained approximately 6,000 images from 10 monkeys, and the test set contained approximately 8,000 images from 3 monkeys. Note that this embodiment did not use the same macaques for training and testing, making it highly robust to different scenarios or different macaques.

[0043] In this embodiment, a threshold is set for the intersection-union ratio (IUU) of the manually annotated ground truth bounding box and the predicted bounding box output by the YOLO network for each test image. If the IUU is greater than the threshold, the detection is considered successful; otherwise, the detection fails. Finally, the proportion of successfully detected images out of all test images is the success rate.

[0044] The specific implementation process and results of this embodiment are as follows: Figure 6 As shown, Figure 6 The first row contains video frame images extracted from the macaque video at fixed intervals; in this embodiment, one image is taken every 5 frames.Figure 6 The second row shows the feature maps extracted from these image frames by the backbone; brighter areas in the map indicate higher activation. Figure 6 The third row shows the positions of the predicted bounding boxes output by the network on the feature map.

[0045] Finally, this embodiment also applies the above test set to the existing YOLOv5 and YOLOv7 (… Figure 3 Success rate experiments were conducted on YOLOv5, Faster-RCNN, RetinaNet, and MonkeyTrail, as shown in the figures. The results showed that YOLOv5 achieved a success rate of 64.6%, the existing YOLOv7 achieved 76.8%, Faster-RCNN achieved 63.8%, RetinaNet achieved 71.0%, and MonkeyTrail achieved 58.9%. In contrast, the improved YOLOv7 of this invention achieved a success rate of 83.1% in animal target motion detection. This invention addresses the problem of the high time and cost associated with manually reviewing large amounts of macaque video data for analysis and recording in current behavioral experiments.

Claims

1. A macaque detection method based on a YOLOv7 network and a Deepsort network, characterized in that: The method comprises the following steps: Step S1, inputting a video sequence into a network The video frames of the video are captured by using an opencv toolkit and frame images are returned, all frame images are respectively subjected to resize processing in the order of the original video sequence, and the frame images subjected to the resize processing are sequentially input into a YOLO network; each BConv layer in the P2 layer in the backbone layer of the YOLOv7 network comprises a ShiftConv convolutional layer, a BN layer and an activation function; the ShiftConv convolutional layer is a TSM module, the slice size of the ShiftConv convolutional layer is 4, the exchange ratio is 1 / 4, and the original input dimension and output dimension are kept unchanged, and the spatiotemporal information of the target video sequence is extracted through the ShiftConv convolutional layer; Step S2, extracting features of the input image through the YOLOv7 network For the frame images input into the backbone layer, the ShiftConv layer selects a plurality of frame images to exchange channel information according to the set batch size, then the head layer of the YOLOv7 network outputs three layers of feature maps with different sizes, and finally the prediction result is obtained by predicting the bounding box on the processed feature map; Step S3, establishing a rhesus monkey motion trajectory path The prediction result is input into a Deepsort network again to assign an id to each target, then a trajectory path is individually established for the bounding box bbox of each target, then the bbox sequence is connected and the abnormal bbox far away from the trajectory path is removed, and finally the Deepsort network outputs the target bbox prediction frame of each frame image in sequence, thereby obtaining a prediction frame sequence; Step S4, detecting and processing missing values of the prediction frame sequence obtained in step S3, if there is no prediction frame in the frame image, the linear interpolation method is used to fill the prediction frame of the frame image; at the same time, according to the coordinate and speed information of the x direction and the y direction of the prediction frame, it is judged whether there is a false detection, if there is a false detection, the false detection result is deleted and the value is filled by using the linear interpolation method.

2. The macaque detection method based on YOLOv7 network and Deepsort network according to claim 1, characterized in that: The backbone layer in the YOLOv7 network in step S1 comprises a plurality of BConv layers, an E-ELAN layer and an MPConv layer, and the SiLU function is used as the activation function in the YOLOv7 network.

Images