A method and related equipment for tracking target animal motion based on video sequence

Through the Deep SORT algorithm combining the depth information of binocular camera and YOLOv4-Tiny model, the problem of difficulty in tracking under large size and occlusion conditions in the prior art is solved, and efficient and accurate analysis of open field animal behavior is achieved.

CN115063447BActive Publication Date: 2024-12-03ZHONGBEI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210639984.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-08
Publication Date
2024-12-03
Estimated Expiration
2042-06-08

AI Technical Summary

Technical Problem

In the prior art, the open field animal behavior analysis method based on convolutional neural networks has a low analysis accuracy due to the large size of the neural network model, and it is impossible to complete the target tracking task when multiple mice accumulate to produce occlusion.

Method used

Images are acquired by binocular cameras, target detection is performed through the YOLOv4-Tiny model, and combined with the depth information input, the improved Deep SORT algorithm is used to achieve the tracking of the motor behavior of the target animal.

Benefits of technology

While ensuring analysis accuracy, the volume of the neural network model is reduced, target tracking under occlusion conditions is achieved, and analysis efficiency and accuracy are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115063447B_ABST
    Figure CN115063447B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for tracking the motion of a target animal based on a video sequence and related equipment. The method comprises: extracting a first image and a second image captured by a binocular camera, performing enhancement processing to obtain an enhanced third image, inputting the enhanced third image into an optimized YOLOv4‑Tiny model to obtain a parameter model, and annotating the second image to obtain a first annotation file; preprocessing the fourth image captured and inputting the fourth image into the parameter model to obtain a detection frame; matching the pixels of the fourth images captured separately to obtain a fifth image, and matching the fifth image with the detection frame, and then inputting the successfully matched fifth image and the detection frame into an improved Deep SORT algorithm model. By optimizing the YOLOv4‑Tiny training model, the obtained parameter model is used to jointly output the motion behavior information of the target object, so as to achieve a lightweight neural network model while ensuring the analysis accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of animal research systems, and in particular to a method for tracking the motion of a target animal based on a video sequence and related equipment. Background Art

[0002] Existing video-based open field animal (open field experiment, also known as open box experiment, is a method to evaluate the autonomous behavior, exploratory behavior and tension of experimental animals in a novel environment. The animals studied are open field animals, mainly mice) behavior analysis methods usually acquire images through a single camera, and then input them into a large-scale convolutional neural network to extract appearance features, perform target detection, and use a combination of a Kalman filter (Kalman filter is an algorithm that uses a linear system state equation to optimally estimate the system state through system input and output observation data) and a Hungarian matching algorithm (the Hungarian algorithm solves an allocation problem. In the main step of multi-target tracking, the similarity is calculated to obtain a similarity matrix between the previous and next frames. The Hungarian algorithm solves the target that truly matches the previous and next frames by solving this similarity matrix) SORT method (SORT (Simple Online and Realtime Tracking) algorithm: a very simple, effective and practical multi-target tracking algorithm) to achieve animal target tracking. Although compared with traditional computer vision methods such as particle filtering, moving target modeling or mean shift, the convolutional neural network-based method has good model generalization and high accuracy, the network model (large-scale convolutional neural network) used is large, often hundreds of MB in size, and often requires a high-performance GPU for inference operations, resulting in extremely low frame rates during real-time analysis. At the same time, this method of obtaining video data through a single camera is almost unable to complete the task of target tracking when multiple mice are piled up and blocked.

[0003] Therefore, the market urgently needs an open-field animal behavior analysis method that can continue to complete target tracking tasks under occlusion conditions, while ensuring accuracy, with a model size as small as possible and low equipment requirements.

[0004] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The main purpose of the present invention is to provide a method and related equipment for tracking the motion of a target animal based on a video sequence, aiming to solve the problems in the prior art of large size of neural network models and low analysis accuracy when analyzing the behavior of animals in open fields.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for tracking the motion of a target animal based on a video sequence, the method comprising the following steps:

[0008] Extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain enhanced third images; wherein all images in the third image set are the third images;

[0009] Performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box;

[0010] Inputting the enhanced third image and the first annotated file simultaneously into the optimized YOLOv4-Tiny model for training to obtain a parameter model;

[0011] After acquiring the fourth image captured by the calibrated binocular camera, preprocessing the fourth image to obtain a preprocessed fourth image, and inputting the preprocessed fourth image into the parameter model to obtain a detection frame of the target object and a confidence level of the target object;

[0012] After performing a pixel matching operation on two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, the depth of the pixels is calculated to obtain a fifth image; wherein the two images captured by the left and right cameras are both referred to as the fourth images;

[0013] The detection box is matched with the bounding box in the fifth image, and the successfully matched fifth image, the confidence level and the detection box are input into an improved Deep SORT algorithm model, and the improved Deep SORT algorithm model predicts and outputs motion behavior information of the target object.

[0014] In the target animal motion tracking method based on video sequences, the step of extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain the enhanced third image also includes:

[0015] The binocular camera is calibrated using a single-plane chessboard camera calibration method to obtain an internal and external parameter transformation matrix and a lens distortion coefficient;

[0016] The binocular camera is calibrated using the lens distortion coefficient.

[0017] In the target animal motion tracking method based on video sequences, the steps of extracting a plurality of first images from a first image set and a plurality of second images from a second image set captured by a calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain an enhanced third image specifically include:

[0018] Using the calibrated binocular camera to photograph the target object to obtain the second image set, and extracting the same number of the first images and the second images from the first image set and the second image set in a uniform random sampling manner to obtain the third image set;

[0019] All the third images in the third image set are enhanced by translation, rotation and noise addition to obtain enhanced third images.

[0020] In the method for tracking target animal motion based on video sequences, the step of performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box specifically includes:

[0021] Classifying and frame processing the second image to obtain the second image with the boundary of the target object;

[0022] Using labelimg software, the boundary of the target object is marked with a rectangular box to obtain a first marking file with the boundary box of the target object.

[0023] In the method for tracking target animal motion based on video sequences, the step of simultaneously inputting the enhanced third image and the first annotation file into the optimized YOLOv4-Tiny model for training to obtain a parameter model specifically includes:

[0024] After replacing the backbone feature extraction network of the original YOLOv4-Tiny model with the Mobilenetv3 structure, the number of layers of the Mobilenetv3 structure is increased, and the inverted residual structure is added to obtain the optimized YOLOv4-Tiny model;

[0025] The enhanced third image and the first annotation file are trained using the optimized YOLOv4-Tiny model to obtain the parameter model.

[0026] In the target animal motion tracking method based on video sequence, the step of performing pixel matching operation on two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time and then calculating the depth of the pixels to obtain the fifth image specifically includes:

[0027] After preprocessing the two preprocessed fourth images captured by the left and right cameras at the same time, pixel matching is performed on the two preprocessed fourth images according to the internal and external parameter transformation matrix;

[0028] The depth of each pixel is calculated according to the matching result to obtain all the fifth images; wherein the fifth image is a depth map.

[0029] In the target animal motion tracking method based on video sequences, the step of matching the detection frame with the bounding box in the fifth image, and inputting the successfully matched fifth image, the confidence level and the detection frame into the improved Deep SORT algorithm model, wherein the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object specifically includes:

[0030] The ShuffleNetv2 network is used to replace the ReID network in the original Deep SORT algorithm model to obtain the improved Deep SORT algorithm model;

[0031] Matching the detection frame with the bounding box in the fifth image, and creating a tracker in the improved Deep SORT algorithm model using the successfully matched fifth image and the detection frame;

[0032] After initializing the Kalman filter in the Deep SORT algorithm model, the Kalman filter is used to predict the state of the target object, and a prediction frame is obtained and saved in the tracker;

[0033] Filtering out the detection frames whose confidence is less than or equal to a preset threshold, and performing threshold matching and non-maximum suppression operations on the tracker and the filtered detection frames;

[0034] The trackers that have successfully matched the threshold value of the determined state are subjected to a cascade matching operation, and the Kalman filter is updated with the trackers that have successfully matched the cascade matching, so that the Kalman filter outputs the motion behavior information of the target object.

[0035] A target animal motion tracking system based on a video sequence, the target animal motion tracking system based on a video sequence comprising:

[0036] An image processing module, used for extracting a plurality of first images in the first image set and a plurality of second images in the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain an enhanced third image; wherein all images in the third image set are the third images;

[0037] A data annotation module, used to perform data annotation processing on the second image to obtain a first annotation file with a target object boundary box;

[0038] A parameter model building module, used for simultaneously inputting the enhanced third image and the first annotation file into the optimized YOLOv4-Tiny model for training to obtain a parameter model;

[0039] A training result prediction module, used for collecting a fourth image captured by the calibrated binocular camera, performing a preprocessing operation on the fourth image to obtain a preprocessed fourth image, and inputting the preprocessed fourth image into the parameter model to obtain a detection frame of the target object and a confidence level of the target object;

[0040] A depth map acquisition module, used for performing a pixel matching operation on two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, and calculating the depth of the pixels to obtain a fifth image; wherein the two images captured by the left and right cameras are both referred to as the fourth images;

[0041] The motion behavior information output module is used to match the detection box with the bounding box in the fifth image, and input the successfully matched fifth image, the confidence level and the detection box into the improved Deep SORT algorithm model, and the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object.

[0042] A controller comprises: a memory, a processor and a target animal motion tracking program based on a video sequence stored in the memory and executable on the processor, wherein the target animal motion tracking program based on a video sequence, when executed by the processor, implements the steps of the target animal motion tracking method based on a video sequence as described above.

[0043] A computer-readable storage medium stores a target animal motion tracking program based on a video sequence, and when the target animal motion tracking program based on a video sequence is executed by a processor, the steps of the target animal motion tracking method based on a video sequence are implemented.

[0044] Compared with the prior art, the present invention provides a method for tracking the motion of a target animal based on a video sequence and related equipment. The method for tracking the motion of a target animal based on a video sequence comprises the following steps: extracting a plurality of first images from a first image set and a plurality of second images from a second image set captured by a calibrated binocular camera, combining the plurality of first images and second images to obtain a third image set, performing enhancement processing on all third images in the third image set to obtain an enhanced third image; wherein all images in the third image set are the third images; performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box; and simultaneously inputting the enhanced third image and the first annotation file into an optimized YOLOv The YOLOv4-Tiny model is trained to obtain a parameter model; after collecting the fourth image captured by the calibrated binocular camera, the fourth image is preprocessed to obtain the preprocessed fourth image, and is input into the parameter model to obtain the detection frame of the target object and the confidence of the target object; after pixel matching operation is performed on the two preprocessed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, the depth of the pixel is calculated to obtain the fifth image; wherein, the two images captured by the left and right cameras are both called the fourth images; the detection frame is matched with the bounding box in the fifth image, and the matched fifth image, the confidence and the detection frame are input into the improved Deep SORT algorithm model, and the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object. By optimizing the YOLOv4-Tiny training model and inputting the depth map after the detection frame and the depth map are successfully matched into the improved Deep SORT algorithm model, the neural network model can be made lightweight while ensuring the analysis accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a preferred embodiment of a method for tracking target animal motion based on a video sequence provided by the present invention;

[0046] Figure 2 A flowchart of step S100 in a preferred embodiment of the method for tracking the motion of a target animal based on a video sequence provided by the present invention;

[0047] Figure 3 A flowchart before step S100 in a preferred embodiment of the method for tracking target animal motion based on video sequences provided by the present invention;

[0048] Figure 4 A chessboard diagram provided in a preferred embodiment of the present invention;

[0049] Figure 5A flowchart of step S200 in a preferred embodiment of the method for tracking the motion of a target animal based on a video sequence provided by the present invention;

[0050] Figure 6 A flowchart of step S300 in a preferred embodiment of the method for tracking the motion of a target animal based on a video sequence provided by the present invention;

[0051] Figure 7 The structure diagram of the original Yolov4-Tiny model provided in the preferred embodiment of the present invention;

[0052] Figure 8 The optimized Yolov4-Tiny model structure diagram provided in the preferred embodiment of the present invention;

[0053] Fig. 9 A flowchart of step S400 in a preferred embodiment of the method for tracking the motion of a target animal based on a video sequence provided by the present invention;

[0054] Fig.10 A flowchart of step S500 in a preferred embodiment of the method for tracking the motion of a target animal based on a video sequence provided by the present invention;

[0055] Fig.11 A flowchart of step S600 in a preferred embodiment of the method for tracking the motion of a target animal based on a video sequence provided by the present invention;

[0056] Fig.12 The relationship between the three states and the transition states of the tracker provided in the preferred embodiment of the present invention;

[0057] Fig.13 A schematic diagram of the ShuffleNetv2 network structure provided in a preferred embodiment of the present invention;

[0058] Fig.14 A flow chart of the new Deep SORT algorithm model provided by the present invention outputting the motion behavior information of the target object;

[0059] Fig.15 A schematic diagram of the steps of outputting the cost matrix C provided in a preferred embodiment of the present invention;

[0060] Fig.16 A schematic diagram of the steps of outputting the result B matrix provided in a preferred embodiment of the present invention;

[0061] Fig.17 A schematic diagram of the process of dividing the matched set and the unmatched set provided by the present invention;

[0062] Fig.18A simplified diagram of the entire workflow of the new Deep SORT algorithm model provided by the present invention;

[0063] Fig.19 The principle block diagram of the target animal motion tracking system based on video sequence provided by the present invention;

[0064] Fig. 20 A schematic diagram of the operating environment of the controller provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0066] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0067] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0068] The present invention provides a method for tracking the motion of a target animal based on a video sequence and related equipment. In the present invention, the enhanced third image and the first annotation file are simultaneously input into an optimized YOLOv4-Tiny model for training to obtain a parameter model; the pre-processed fourth image is input into the parameter model to obtain a detection frame and a confidence level; and the successfully matched fifth image, the confidence level and the detection frame are input into an improved Deep SORT algorithm model to output the motion behavior information of the target object, thereby effectively achieving the lightweighting of the neural network model while ensuring the analysis accuracy.

[0069] The following describes the design of the target animal motion tracking method based on the video sequence through specific exemplary embodiments. It should be noted that the following embodiments are only used to explain the technical solution of the invention and are not specifically limited:

[0070] See also Figure 1 The present invention provides a method for tracking the motion of a target animal based on a video sequence, and the method for tracking the motion of a target animal based on a video sequence comprises the following steps:

[0071] S100. Extract multiple first images from the first image set and multiple second images from the second image set captured by the calibrated binocular camera, combine the multiple first images and second images to obtain a third image set, and enhance all third images in the third image set to obtain enhanced third images; wherein all images in the third image set are the third images.

[0072] Specifically, for target detection, it can be divided into three major processes: advance preparation, model training and model prediction, and the three processes are independent of each other: among them, advance preparation includes: data set acquisition, data annotation and camera calibration; and the data set acquisition process is: extracting multiple first images from the first image set publicly available on the Internet (such as the image data set publicly available on the Internet) and multiple second images from the second image set (training original image set) taken by the calibrated binocular camera in a predetermined manner, thereby obtaining a third image set (training image set), selecting targeted data for training the model, which is conducive to improving the accuracy of the trained model. Among them, the binocular camera can be installed on the top of the experimental box.

[0073] For further information, see Figure 2 S100, the step of extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain an enhanced third image specifically includes:

[0074] S110, photographing the target object using the calibrated binocular camera to obtain the second image set, and extracting the same number of the first images and the second images from the first image set and the second image set in a uniform random sampling manner to obtain the third image set;

[0075] S120, performing enhancement processing on all the third images in the third image set by means of translation, rotation and adding noise to obtain enhanced third images.

[0076] Among them, data enhancement (specifically image data enhancement): Deep neural networks are driven by data. When the quality can be guaranteed, the larger the data, the better the generalization performance and performance of the model. However, data annotation is time-consuming and laborious, and it is often not easy to collect too much. By using existing data and processing data such as flipping, more pictures can be created, thereby improving the accuracy and generalization ability of the network. For example, the imgaug open source library can be used for image data enhancement. It can generate a set of new pictures by inputting pictures, and can change pictures through different strategies (selecting, translating, flipping, adding noise, scaling, brightness contrast, chromaticity saturation transformation, etc.) to achieve the purpose of data enhancement.

[0077] Specifically, first, the target object is photographed using a calibrated binocular camera to obtain the second image set (the second image in the second image set is a training original image, which is an RGB image), and then, the same number of the first images and the second images are extracted from the first image set (for example, an image data set publicly available on the Internet, if the target object is a mouse in an open field experiment, the first image set specifically refers to a mouse open field experiment gait tracking data set) and the second image set in a preset ratio by uniform random sampling to obtain the third image set, for example, n first images and n second images are extracted in a ratio of 8:1:1 by uniform random sampling, and are correspondingly divided into a training set, a test set and a validation set in the third image set, wherein the data set is used to train the weight parameters of the neural network model, and the test set and the validation set are used to evaluate the model performance and adjust the optimization model; secondly, the divided third image is translated, rotated, and noise is added to obtain an enhanced third image, and the data set is enhanced to improve the generalization performance of the trained model so that it can be applied to different experimental scenarios.

[0078] Among them, for RGB images, the RGB color mode is a color standard in the industry. Various colors are obtained by changing the three color channels of red (R), green (G) and blue (B) and superimposing them on each other. RGB represents the colors of the three channels of red, green and blue. This standard covers almost all colors that can be perceived by human vision and is one of the most widely used color systems.

[0079] For further information, see Figure 3 S100, extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain an enhanced third image, further comprising:

[0080] S10, calibrating the binocular camera using a single-plane chessboard camera calibration method to obtain an internal and external parameter transformation matrix and a lens distortion coefficient;

[0081] S20: calibrating the binocular camera using the lens distortion coefficient.

[0082] First of all, you need to understand why camera calibration is needed?

[0083] One reason is that the degree of distortion of each lens during the production and assembly process is different. Through camera calibration, this lens distortion can be corrected to generate a corrected image - corrected lens distortion; another reason is to establish a camera imaging geometry model based on the calibrated camera parameters, and reconstruct the three-dimensional scene from the obtained image. Specifically: when taking pictures with a camera, some spatial information (such as distance and size, etc.) is obtained from the photo by using a two-dimensional image to obtain three-dimensional information; when taking pictures, the spatial object information is converted into a two-dimensional image through the camera, and this process is originally irreversible. However, if a mathematical model of the camera can be found, the original three-dimensional information can be reversed from the two-dimensional image + model, and the calibration operation is to find this model.

[0084] Then, after calibrating the camera, we can get: the camera's intrinsic parameter matrix A (dx, dy, r, u, v, f), extrinsic parameter matrix [R|T] and distortion coefficients [k1, k2, k3, ~, p1, p2, ~]. The meaning of each element of the intrinsic parameter matrix is: the physical size dx and dy of a pixel, the focal length f, the distortion factor r of the image physical coordinates, the vertical and horizontal offsets u and v (in pixels) of the image origin relative to the optical center imaging point; the extrinsic parameter matrix: the rotation R and translation T matrix of the world coordinate system to the camera coordinate system; the distortion coefficient: including the radial distortion coefficients k1, k2, k3, ~, and the tangential distortion coefficients p1, p2, ~ of the camera.

[0085] Specifically, the binocular camera is calibrated using a single-plane chessboard camera calibration method to obtain an internal and external parameter transformation matrix (including the camera's internal parameter matrix A(dx, dy, r, u, v, f) and external parameter matrix [R|T]) and lens distortion coefficients [k1, k2, k3, ~, p1, p2, ~].

[0086] The binocular camera (camera) is calibrated by chessboard calibration method, which is performed on matlab software. The specific principle and process are as follows: the chessboard is a calibration board composed of black and white squares, which is used as a calibration object for camera calibration (an object mapped from the real world to the digital image). The reason why the chessboard is used as a calibration object is that the flat chessboard pattern is easier to handle than a complex three-dimensional object, but at the same time, a two-dimensional object will lack some information relative to a three-dimensional object, so the image can be captured by changing the orientation of the chessboard multiple times in order to obtain richer coordinate information.

[0087] Next, we will perform a series of transformations on the rigid body, so that it can be affine transformed and projected from the world coordinate system, and finally obtain discrete image points in the pixel coordinate system. In the process, various parameter matrices will be gradually introduced. The calibration images need to be taken at different positions, angles and postures using a calibration plate. At least 3 images are required, and 10 to 20 images are appropriate. The calibration plate needs to be a chessboard composed of black and white rectangles, and the production accuracy requirements are high, such as Figure 4 shown.

[0088] The calibration process is as follows: first, print a chessboard and stick it on a plane as a calibration object; then, by adjusting the direction of the calibration object or the camera, take some photos of the calibration object in different directions; then, use the corner point extraction algorithm to obtain the grid points of the chessboard plane and estimate the five internal parameters and six external parameters under the ideal distortion-free condition; then, use the least squares method to estimate the distortion coefficient under actual radial distortion, and finally, use the maximum likelihood method to optimize the estimation result to improve the estimation accuracy.

[0089] Then, the binocular camera is calibrated using the lens distortion coefficient obtained after calibrating the binocular camera, so that the accurate second image captured by the calibrated binocular camera can be used, that is, the second image with high precision and high quality can be obtained.

[0090] For further information, please refer to Figure 1 S200: performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box; wherein the first annotation file with the target object boundary box refers to an .xml file containing the object center position and the object boundary position.

[0091] Specifically, in order to generate a first annotation file with a target object bounding box, it is necessary to perform data annotation processing on the second image captured by the calibrated binocular camera, wherein data annotation specifically refers to the annotation of image data in target detection, and data annotation in target detection refers to the process of processing image data through three steps of classification, framing and annotation to mark the features of the object as the basic material for machine learning.

[0092] For further information, see Figure 5 S200, the step of performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box specifically includes:

[0093] S210, classifying and frame processing the second image to obtain the second image with the boundary of the target object;

[0094] S220, using labelimg software, annotate the boundary of the target object with a rectangular box to obtain a first annotation file with the boundary box of the target object. The rectangular box is used to enclose the target during operations such as data annotation, model training, and model prediction. In the computer, the rectangular box data is saved in the format of (x, y, w, h), where x is the horizontal coordinate of the upper left corner of the target, y is the vertical coordinate of the upper left corner of the target, w is the width of the target, and h is the height of the target.

[0095] Specifically, when target detection is required, first create a data set folder, put several pictures of the target to be detected in it (classification processing), then use labelimg software to open the folder, and perform the following operations: first open the file, create a rectangular frame (picture frame processing), and obtain the second image with the boundary of the target object, then place the object completely in the frame while the frame fits the target object as close as possible, and finally, indicate the category (labeling processing) and save it, so as to obtain the first labeled file with the boundary box of the target object.

[0096] For further information, please refer to Figure 1, S300, inputting the enhanced third image and the first annotation file into the optimized YOLOv4-Tiny model for training to obtain a parameter model.

[0097] Among them, the Yolo (You Only Look Once) algorithm is a regression-based target detection algorithm. The Yolo algorithm is characterized by extremely fast computing speed while ensuring accuracy. Yolov4-Tiny is a simplified version of the fourth-generation Yolo algorithm Yolov4. Compared with Yolov4, it has a smaller model size and faster computing speed while reducing some of its accuracy, making it suitable for deployment on various embedded devices.

[0098] Specifically, in a broad sense, machine learning is a method that can give machines the ability to learn, so that it can complete functions that cannot be completed by direct programming. But in a practical sense, machine learning is a method of using data to train a model and then using the model for prediction. Among them, machine learning is divided into supervised learning, unsupervised learning, weakly supervised learning and reinforcement learning according to whether the label information of the training sample is unknown or known. Deep learning is part of supervised learning. The so-called supervised learning is to train an optimal model through existing training samples so that the optimal model can be used for data prediction and analysis later; then the enhanced third image and the first annotation file are simultaneously input into the optimized YOLOv4-Tiny model for training to obtain a parameter model, which belongs to the process of training an optimal model through existing training samples, that is, the process of model training. The training process is mainly to determine the value of the weight in the YOLOv4-Tiny model. The entire training is end-to-end and can be completed automatically by automated equipment.

[0099] For further information, see Figure 6 S300: simultaneously inputting the enhanced third image and the first annotation file into the optimized YOLOv4-Tiny model for training to obtain a parameter model comprises:

[0100] S310, after replacing the backbone feature extraction network of the original YOLOv4-Tiny model with the Mobilenetv3 structure, increasing the number of layers of the Mobilenetv3 structure, and adding an inverted residual structure, to obtain an optimized YOLOv4-Tiny model;

[0101] S320: Use the optimized YOLOv4-Tiny model to train the enhanced third image and the first annotation file to obtain the parameter model.

[0102] Specifically, the original Yolov4-Tiny network model is as follows, which includes five basic components:

[0103] First, the CBL layer, which is composed of the conv convolution layer, the BN layer and the Leaky Relu activation function; second, the CSP layer, which divides the original input into two branches, performs convolution operations on each branch to halve the number of channels, and then performs Bottleneck*N operations on one branch, and then connects the two branches so that the input and output of Bottlenneck CSP are the same size, so that the model can learn more features; third, the pooling layer, which uses maximum pooling to reduce information redundancy and prevent model overfitting; fourth, up and down sampling: that is, using interpolation algorithms to enlarge or reduce images; fifth, the BackBone backbone network: mainly the connection of the basic components CBL layer, pooling layer, and CSP layer; among them, the original Yolov4-Tiny model structure diagram is as follows Figure 7 shown.

[0104] However, the original Yolov4-Tiny network model has the following main problems in the task of target detection:

[0105] 1. The recognition accuracy for small targets is not high enough; 2. There is a motion blur problem for dense targets.

[0106] The main reason for these problems is that the Yolov4-Tiny model does not extract enough features for small and dense targets. To address this problem, the original Yolov4-Tiny backbone feature extraction network is replaced with the Mobilenetv3 structure. The receptive field is increased by increasing the number of layers of the Mobilenetv3 structure (new backbone feature extraction network), that is, the multi-layer convolution in the YOLOv4-Tiny model is used to increase the receptive field, thereby increasing the feature extraction of small and dense targets. Therefore, the YOLOv4-Tiny model uses a large receptive field for classification based on high-order features in the final classification.

[0107] Among them, the receptive field refers to the input area "seen" by neurons in a neural network. In a convolutional neural network, the calculation of an element on a feature map is affected by a certain area on the input image, and this area is the receptive field of the element.

[0108] At the same time, the actual number of operations of the model is further reduced, and an inverted residual structure is added, so that although the output of the final Yolov4-Tiny model is still the same depth, the internal can be expanded to a high-dimensional feature space, increasing the expressiveness of the nonlinear channel, that is, the Yolov4-Tiny network is stretched to a high dimension at the lowest order possible, thereby ensuring convergence, and then reducing time overhead while being able to extract rich features; Next, the model parameter weights are continuously updated through back propagation and batch stochastic gradient descent until the model loss reaches the minimum, thereby obtaining the optimized YOLOv4-Tiny model (optimal model), and the H-Swish activation function is also introduced, so as to effectively avoid the loss of precision caused by quantization while maintaining accuracy. The structure diagram of the optimized Yolov4-Tiny model is shown below. Figure 8 Then, the optimized YOLOv4-Tiny model is used to train the enhanced third image and the first annotation file to obtain the parameter model (optimal model). In the present invention, the original Yolov4-Tiny network model is improved to obtain a more lightweight, fast and accurate parameter model, so that the parameter model can be used to make a more accurate and fast prediction of the image to be predicted.

[0109] Replace the backbone feature extraction network of the original YOLOv4-Tiny model with the Mobilenetv3 structure, and increase the number of layers of the backbone feature extraction network;

[0110] Increase the feature extraction of small and dense targets, reduce the actual number of model operations, and add an inverted residual structure.

[0111] For further information, please refer to Figure 1 S400, after acquiring the fourth image captured by the calibrated binocular camera, preprocessing the fourth image to obtain the preprocessed fourth image, and inputting the preprocessed fourth image into the parameter model to obtain the detection frame of the target object and the confidence of the target object.

[0112] Among them, confidence is also called reliability, confidence level or confidence coefficient, which means that in the prediction process of target detection, the detected objects will be classified and a column vector of category scores will be output. The column vector contains scores of different categories. The higher the score of a category, the more likely the computer thinks that the object belongs to that category. This score is called detection confidence. In this embodiment, the score indicating that it is the target object is the confidence. For example, the score indicating that it is a mouse. In this example, mice are used as an example of open field animals.

[0113] Specifically, after training an optimal model (the parameter model) through existing training samples, this model is used to map all inputs to corresponding outputs, and a simple judgment is made on the output to achieve the purpose of prediction and classification. It also has the ability to predict and classify unknown data. Here, the preprocessed fourth image is input into the parameter model, and the prediction result is output: the detection box and confidence, that is, the model prediction process.

[0114] For further information, see Fig. 9 S400, after acquiring the fourth image captured by the calibrated binocular camera, preprocessing the fourth image to obtain the preprocessed fourth image, and inputting the preprocessed fourth image into the parameter model, the step of obtaining the detection frame of the target object and the confidence of the target object specifically includes:

[0115] S410, after acquiring the fourth image captured by the calibrated binocular camera, a Gaussian filter operation is performed on the fourth image using the open source OpenCV library to obtain a preprocessed fourth image;

[0116] S420: Input the preprocessed fourth image into the parameter model to obtain the detection frame and confidence level.

[0117] Specifically, the binocular camera has a left camera and a right camera. Generally, the left camera has a high definition and the right camera has a low definition. Both cameras collect RGB data. The RGB data collected by the left and right cameras after correction are referred to as left RGB data and right RGB data below. The left RGB data and the right RGB data are collectively referred to as the fourth image (image to be predicted). Then, both the left RGB data and the right RGB data need to undergo a data preprocessing process, namely: using the open source OpenCV library (OpenCV is a cross-platform computer vision and machine learning software library released based on the Apache2.0 license (open source)) to perform Gaussian filtering on the fourth image, including weighted averaging of the fourth image, etc., to reduce noise in the image, thereby obtaining a preprocessed fourth image.

[0118] Then, the preprocessed fourth image is input into the parameter model to obtain a prediction result, namely, the detection box and the confidence level.

[0119] For further information, please refer to Figure 1, S500, after performing a pixel matching operation on the two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, calculate the depth of the pixels to obtain a fifth image; wherein the two images respectively captured by the left and right cameras are both called the fourth images.

[0120] Specifically, the pixel matching operation is performed on the two pre-processed fourth images taken by the calibrated left and right cameras at the same time, and the depth of the pixels is calculated to obtain the fifth image, so that the depth map can be used as an input parameter in the improved Deep SORT algorithm model. The Deep SORT algorithm is an algorithm for multi-target tracking that is improved on the SORT algorithm.

[0121] For further information, see Fig.10 S500, the step of performing pixel matching operation on the two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, and calculating the depth of the pixels to obtain the fifth image specifically includes:

[0122] S510, after performing a preprocessing operation on two preprocessed fourth images captured by the left and right cameras at the same time, pixel matching is performed on the two preprocessed fourth images according to the internal and external parameter transformation matrix;

[0123] S520. Calculate the depth of each pixel according to the matching result to obtain all the fifth images; wherein the fifth images are depth maps.

[0124] Specifically, the fourth image (RGB image) captured by the binocular camera is first preprocessed to obtain a preprocessed fourth image, and then the world coordinate system in the two preprocessed fourth images captured by the left and right cameras at the same time is converted into a pixel coordinate system using the intrinsic and extrinsic matrix, and then the two preprocessed and coordinate-converted fourth images are matched pixel by pixel, and finally the depth of each pixel is calculated according to the matching result to obtain an input parameter in the improved Deep SORT algorithm model, namely, the fifth image, or the depth map (depth data).

[0125] For further information, please refer to Figure 1 , S600, matching the detection box with the bounding box in the fifth image, and inputting the successfully matched fifth image, the confidence level and the detection box into an improved Deep SORT algorithm model, and the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object.

[0126] Specifically, after calculating the depth of each pixel according to the matching results and obtaining all the fifth images (the depth maps), the detection box is matched with the bounding box in the fifth image, and the successfully matched fifth image, the confidence level and the detection box are input into the improved Deep SORT algorithm model. The improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object, thereby realizing the monitoring and recording of the motion behavior information of the target object.

[0127] For further information, see Fig.11 S600, matching the detection frame with the bounding box in the fifth image, and inputting the successfully matched fifth image, the confidence and the detection frame into the improved Deep SORT algorithm model, wherein the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object, specifically comprising:

[0128] S610, using ShuffleNetv2 network to replace the ReID network in the original Deep SORT algorithm model to obtain an improved Deep SORT algorithm model;

[0129] S620, matching the detection frame with the bounding box in the fifth image, and creating a tracker in an improved Deep SORT algorithm model using the successfully matched fifth image and the detection frame;

[0130] S630, after initializing the Kalman filter in the Deep SORT algorithm model, using the Kalman filter to predict the state of the target object, and obtaining a prediction frame, and saving it in the tracker;

[0131] S640, filtering out the detection frames whose confidence is less than or equal to a preset threshold, and performing threshold matching and non-maximum suppression operations on the tracker and the filtered detection frames;

[0132] S650, performing a cascade matching operation on the trackers that have successfully matched the threshold of the determined state, and using the trackers that have successfully matched the cascade matching to update the Kalman filter, so that the Kalman filter outputs the motion behavior information of the target object.

[0133] Among them, in order to introduce non-maximum suppression, the concept of intersection-over-union ratio is first introduced. The intersection-over-union ratio (IOU) function is used to calculate the ratio of the intersection and union of two bounding boxes. The union of two bounding boxes is this area, that is, the area containing two bounding boxes, and the intersection is this smaller area, then the intersection-over-union ratio is the size of the intersection divided by the area of ​​the union, so IOU measures the relative size of the overlap of two bounding boxes. And the effect of non-maximum suppression is that when the target object is detected, the same object may be detected multiple times, resulting in many bounding boxes, but the operation of non-maximum suppression can ensure that the algorithm detects each object only once, and the non-maximum value means that only the classification result with the highest probability is output. Although the suppression is very close, it is not the largest other prediction result, so this method is called non-maximum suppression.

[0134] The significance of (IOU) threshold matching is that for different detection tasks, if the scales of the targets to be detected are not much different, that is, most of the targets in the data set are of the same scale, the IOU threshold can be appropriately lowered for selection to achieve the maximum extraction of small target features. However, in practical applications, detection in the same scene cannot only include targets of a single scale. There are situations where the scales of different targets vary greatly. If the IOU threshold is fixed for unified detection and screening, it will cause sample imbalance problems, and small target features are very likely to be discarded by the strict IOU threshold. Therefore, setting a dynamic IOU threshold as a detection for targets of different scales is more universal, and it is dynamically adjusted according to different sample numbers. When the number of negative samples is too high, the IOU threshold is continuously increased to balance the number of samples, avoiding missed detections caused by directly setting too high an IOU threshold, and the trained model has stronger generalization.

[0135] The Kalman filter is a standard Kalman filter with a constant speed motion and a linear observation model. The two core steps of the Kalman filter algorithm adopted by the Kalman filter are prediction and update; wherein, prediction is to predict the state of the target object at time t based on the state of the target object at time t-1.

[0136] The forecast is mainly divided into two parts:

[0137]

[0138] In formula (1), x is the mean of track at time t-1, F is called the state transfer matrix, and this formula predicts x' (mean vector) at time t; in formula (2), P is the covariance of track at time t-1, Q is the noise matrix of the system, which represents the reliability of the entire system and is generally initialized to a very small value. This formula predicts P' (covariance matrix) at time t.

[0139] The update is based on the detection result (measurement value) at time t and the state of the target at time t (prediction value) predicted according to the tracking trajectory, to obtain a more accurate result (state) at time t.

[0140]

[0141] In formula (3), z is the mean vector of detection (the mean vector is calculated in the initiate() function), which does not include the speed change value, that is, z = [x, y, a, h]. H is called the measurement matrix, which maps the mean vector x' of track to the detection space. This formula calculates the mean error y between detection and track. The value of the measurement matrix H is as follows:

[0142]

[0143] In formula (4), R is the noise matrix of the detector, which is a 4x4 diagonal matrix. The values ​​on the diagonal are the two coordinates of the center point and the noise of width and height, respectively. It is initialized with arbitrary values. Generally, the noise of width and height is set to be greater than the noise of the center point. This formula first maps the covariance matrix P' at the predicted time t to the detection space, and then adds the noise matrix R; in formula (5), the Kalman gain K is calculated, and the Kalman gain is used to estimate the importance of the error; in formulas (6) and (7), the updated mean vector x and covariance matrix P are calculated.

[0144] Specifically, the Kalman filter in the original Deep SORT algorithm model uses an 8-dimensional spatial quantity (u, v, r, h, x', y', r', h') to describe the center position, aspect ratio, height, speed information and observation variables of the target frame of the tracked target object at a certain moment; however, the depth data is added in the present application to obtain a 9-dimensional spatial quantity (u, v, r, h, z, x', y', r', h', z'); wherein, (u, v) is the horizontal and vertical coordinates of the center position of the tracker, r is the aspect ratio of the tracker, h is the height of the tracker, z is the depth information of the center position of the tracker, (x', y', r', h') is the movement speed of the tracker in the x direction, the movement speed of the tracker in the y direction, the change in the aspect ratio of the tracker relative to time, the change in the height of the tracker relative to time, and z' is the change in the depth of the center position of the tracker relative to time;

[0145] Among them, the tracker in this application is generated by initializing the Kalman filter in the new Deep SORT algorithm model, and its motion state (u, v, r, h, z, x', y', r', h', z') and covariance information are updated through the Kalman filter, and the tracker contains the prediction information of the target object (the prediction box, etc.).

[0146] The tracker has three states: confirmed, unconfirmed, tentative, and deleted. The relationship and transition between the three states are as follows: Fig.12 As shown in the figure, n_init and max_age are set by yourself, n_init is usually 3, i.e. the first three frames; max_age is the set maximum storage time. If the tracker does not match any previous tracker after this time, it is considered that the target object has left the camera; the matching in the figure includes: the matching of the tracker with the detection result and the matching of the tracker with the previous tracker.

[0147] In this application, the ShuffleNetv2 network is used to replace the ReID network in the original Deep SORT algorithm model to obtain an improved Deep SORT algorithm model (new Deep SORT algorithm model), wherein the ShuffleNetv2 network structure is as follows: Fig.13 As shown, the detailed process of using the new Deep SORT algorithm model to output the motion behavior information of the target object is as follows: Fig.14 As shown:

[0148] Step 1: Using the detection frame of the current target object, the confidence of the detected object and the depth data of the center of the detection frame (the depth map) as inputs of the new Deep SORT algorithm model;

[0149] Step 2: Match the detection box of the i-th frame output by the parameter model with the bounding box in the fifth image (the depth data) after pixel matching, and use the successfully matched fifth image and the detection box to create the corresponding tracker (Tracks) in the new Deep SORT algorithm model, obtain the motion state (u, v, r, h, z, x', y', r', h', z') and covariance information (including covariance matrix P, observation matrix H, measurement state covariance matrix R) of the tracker, and initialize it. At this time, the tracker is in an unknown state: tentative;

[0150] Then, the covariance matrix P, observation matrix H, measurement state covariance matrix R and Kalman gain matrix K of the Kalman filter are initialized, and the initialized Kalman filter is used to predict the state of the target object, and the prediction frame is obtained and saved in the tracker, that is, the tracker contains the prediction frame of the target object, etc.

[0151] Secondly, according to the input confidence, the detection box whose confidence is less than or equal to a preset threshold (for example, 0.7) is filtered out to obtain the detection results whose confidence is greater than the preset threshold and has not been matched with any tracker before, as well as all detection results (the detection results include the detection box, etc.) whose confidence of the input image is greater than the preset threshold on consecutive n_init frames (for example, the first three frames); wherein the detection box is the image box of the current target object detected in the RGB image of the i-th frame output after the RGB image of the i-th frame is input into the optimized Yolov4-Tiny network, including (x, y, w, h) information, reflecting the current information.

[0152] Step 3: Perform IOU threshold matching and NMS non-maximum suppression on the predicted box predicted by the tracker of the i-1th frame and the current detection result of the i-th frame (the detection box), and calculate the cost matrix (the calculation method is 1-IOU) through the result of IOU threshold matching.

[0153] Step 4: Use all calculated cost matrices as the input of the Hungarian algorithm, and obtain the linear matching result of the tracker after Hungarian matching.

[0154] The linear matching results include:

[0155] (1) Unmatched Tracks: The tracker at this time cannot be matched with the previous tracker. If the tracker is in an unknown state and is outside the continuous n_init frame, the tracker is deleted (that is, it is in the deleted state, because this track is in an unknown state. If it is in a confirmed state, it must reach a certain number of times (the default is 30 times) before it can be deleted); if the tracker is in a confirmed state and there is no match for the continuous max_age frame, the tracker enters the deleted state;

[0156] (2) Unmatched Detections: If the current detection result does not match any of the previous trackers, a new tracker is initialized for the detection result and tracking is restarted. Go to step 6.

[0157] (3) The tracker successfully matches the previous tracker: This indicates that the tracking of the i-1th frame and the i-th frame is successful. The apparent features of the successfully matched tracker are calculated and input into the Kalman filter update to update its state. At this time, jump to step 6;

[0158] Step 5: Repeat steps 3 and 4 until a confirmed tracker appears;

[0159] Step 6: Kalman filter update: The input is the motion information and covariance information of the confirmed tracker that has successfully matched (threshold) and the newly assigned tracker that has not successfully matched (threshold). The Kalman filter updates the gain matrix of the Kalman filter through the input and outputs the prediction of the tracker motion information and covariance information;

[0160] At this time, for the tracker that has successfully matched the (threshold) and is in the confirmed state, further enter the cascade matching (previously, each time the tracker matches, the appearance features and motion information of the detection result will be saved. By default, the first 100 frames are saved, and the appearance features and motion information of the tracker are used to perform cascade matching with the detection result. This is because the confirmed tracker is more likely to match the detection result), and jump to step 7;

[0161] For the newly assigned tracker that has not been successfully matched (threshold), it is input into the IOU threshold matching and continues the IOU threshold matching, that is, jump to step 3;

[0162] Step 7: Cascade matching: The input of cascade matching is the motion information, appearance characteristics and covariance information of the successfully matched and confirmed tracker and the (x, y, w, h) information of the detection result. The output of cascade matching has the following three possibilities:

[0163] (1) The successfully matched and confirmed tracker (Tracks matching) is input into the Kalman filter update to update the variables of its corresponding tracker;

[0164] (2) Trackers that do not match any previous trackers (Tracks mismatch);

[0165] (3) Trackers that do not match the current detection results (Detections mismatch);

[0166] For the cascade matching results (2) and (3), continue with IOU threshold matching, i.e. step 3;

[0167] Step 8: Output the motion information of the successfully matched and confirmed tracker, that is, the Kalman filter finally outputs the motion behavior information of the target object.

[0168] Among them, in the Deep SORT algorithm model, the results of Kalman filtering are divided into deterministic and uncertain states, and cascade matching is performed only for the prediction results of the deterministic state.

[0169] When a target is obscured for a long time, the uncertainty of Kalman filter prediction will increase greatly, and the observability in the state space will be greatly reduced. If two trackers compete for the matching right of the same detection result at this time, the track with longer occlusion time will have greater uncertainty in tracking prediction position because the position information has not been updated for a long time, that is, the covariance will be larger. The inverse of the covariance is used in the calculation of the Mahalanobis distance, so the Mahalanobis distance will be smaller, making the detection result more likely to be associated with the track with longer occlusion time. This undesirable effect often destroys the continuity of tracking.

[0170] The core idea of ​​cascade matching is to match the trajectories with the same disappearance time from small to large, which first ensures that the most recently appeared target is given the highest priority and solves the above-mentioned problem. In the final stage of matching, the unconfirmed and unmatched trajectories with age=1 are matched with the detected targets based on IOU, which can alleviate the large changes caused by apparent mutation or partial occlusion. Of course, there are pros and cons. Doing so may also cause some newly generated trajectories to be connected to some old trajectories, but this situation is rare.

[0171] The specific process of cascade matching is as follows:

[0172] In the Kalman filter results, the set of tracking frames in the determined state is recorded as T, and the set of results detected by the current target is recorded as D.

[0173] Step 1: Send the images corresponding to the results in T and D to the appearance feature extraction network, and then calculate the cosine distance of the results calculated by the network, which is used as the cost matrix (C) here. The schematic diagram of the steps for outputting the cost matrix C is as follows Fig.15 As shown, the cost matrix C is used to store the calculation results of the cosine distance between all object tracking i and object detection j.

[0174] Step 2: Calculate the Mahalanobis distance of the detection and prediction results, and then remove the matching items that do not meet the Mahalanobis distance and cosine distance thresholds according to the thresholds obtained from statistics and experiments, and output the result B matrix. The steps for outputting the result B matrix are shown in the figure below: Fig.16 As shown, the B matrix is ​​used to store the judgment of whether all object tracking i and object detection j are associated.

[0175] Step 3: According to the update status of the prediction box (the update status here refers to the time since the last successful match of this prediction box), the newer the prediction box (that is, the shorter the number of frames since the last match), the more priority it has to match the Hungarian algorithm according to the result of C. Finally, the matching set and the unmatched set are divided according to the result of B. The division process is shown in the figure. Fig.17 shown.

[0176] In the present invention, the fifth image that has been successfully matched and the detection frame are first input into the internal Kalman filter of the new Deep SORT algorithm model to initialize it; then, a non-maximum suppression operation is performed, that is, the part with a confidence level less than 0.7 is removed. Next, the new Deep SORT algorithm model matches the positional relationship and appearance feature information between the prediction frame data (prediction frame) output by the Kalman filter and the current detection frame (the detection frame), and determines the degree of association between the prediction frame and the current detection frame based on the matching result, thereby completing the tracking and matching of multiple targets.

[0177] Then, the entire workflow of the new Deep SORT algorithm model is simplified as follows Fig.18 As shown, specifically: in the i-th frame, after the target object to be detected is input, the current appearance feature information is extracted to establish a Kalman filter. In the i+1-th frame, the detection frame data (prediction frame) is predicted by the Kalman filter, and the appearance is matched with the currently extracted appearance features. If the match is successful, the Kalman filter is updated and the subsequent detection of the i+1-th frame is continued. Otherwise, the Kalman filter is reinitialized and the subsequent detection of the i+1-th frame is also continued.

[0178] The above process has achieved matching the detection box with the bounding box in the fifth image, and after inputting the successfully matched fifth image and the detection box into the improved Deep SORT algorithm model, it has achieved outputting the motion trajectory and time node information of different target objects, that is, outputting the motion behavior information of different target objects. By combining the two, the motion speed of the target object, the motion amount in different time periods and the motion trajectory in different time periods can be further calculated and derived.

[0179] The original Deep SORT algorithm uses the ReID domain model to extract appearance features of objects. When running on edge devices, the frame rate will be reduced due to the computing power of the edge devices. In this application, the ShuffleNetv2 network replaces the ReID domain model in the original Deep SORT algorithm model, thereby improving the original Deep SORT algorithm to obtain a new Deep SORT algorithm model, which effectively reduces the amount of calculation while ensuring accuracy as much as possible; and, by fusing the depth information (the depth map or depth data) with the two-dimensional spatial motion information, that is, the depth information of the detected object is also used as the input of the new Deep SORT network. This method solves the problem of tracking loss caused by target occlusion during multi-target tracking. By introducing the depth information of the target animal, when two or more target objects, such as mice, are occluded, the improved Deep SORT algorithm will always retain the tracking information of the mouse that caused the occlusion, and use the change in its depth information and the current amount to match the target object, thereby effectively solving the target tracking problem under occlusion and increasing the prediction accuracy of the Deep SORT algorithm model.

[0180] See also Fig.19The present invention provides a target animal motion tracking system based on video sequences, and the target animal motion tracking system based on video sequences includes: an image processing module 100, a data annotation module 200, a parameter model construction module 300, a training result prediction module 400, a depth map acquisition module 500 and a motion behavior information output module 600; the image processing module 100 is used to extract multiple first images in a first image set and multiple second images in a second image set taken by a calibrated binocular camera, combine the multiple first images and the second images to obtain a third image set, and enhance all third images in the third image set to obtain an enhanced third image; wherein all images in the third image set are the third images; the data annotation module 200 is used to perform data annotation processing on the second image to obtain a first annotation file with a target object boundary box; the parameter model construction module 300 is used to combine the enhanced third image and the The first annotation file is simultaneously input into the optimized YOLOv4-Tiny model for training to obtain a parameter model; the training result prediction module 400 is used to collect the fourth image captured by the calibrated binocular camera, preprocess the fourth image to obtain the preprocessed fourth image, and input it into the parameter model to obtain the detection frame of the target object and the confidence of the target object; the depth map acquisition module 500 is used to perform a pixel matching operation on the two preprocessed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, calculate the depth of the pixels to obtain a fifth image; wherein the two images respectively captured by the left and right cameras are both called the fourth images; the motion behavior information output module 600 is used to match the detection frame with the bounding box in the fifth image, and input the successfully matched fifth image, the confidence and the detection frame into the improved Deep SORT algorithm model, and the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object.

[0181] Specifically, in the present invention, a plurality of first images in a first image set and a plurality of second images in a second image set captured by a calibrated binocular camera are firstly extracted, the plurality of first images and the second images are combined to obtain a third image set, all the third images in the third image set are enhanced to obtain an enhanced third image, and then, the second image is subjected to data annotation to obtain a first annotation file with a target object boundary box, and then, the enhanced third image and the first annotation file are simultaneously input into an optimized YOLOv4-Tiny model for training to obtain a parameter model; furthermore, the preprocessed fourth image is input into the parameter model to obtain a detection box and a confidence level; finally, the successfully matched fifth image, the confidence level and the detection box are input into an improved Deep SORT algorithm model, and the improved Deep After the SORT algorithm model is built, the motion trajectory and time node information of different target objects can be output, that is, the motion behavior information of different target objects can be output. The combination of the two can further calculate and derive the motion speed, motion volume in different time periods and motion trajectory of the target objects in different time periods, thereby effectively achieving not only a smaller model and a faster frame rate while ensuring tracking accuracy, but also easy deployment to various embedded processors, that is, making the neural network model lightweight while ensuring analysis accuracy, and effectively tracking the target when mice are stacked and blocked.

[0182] Furthermore, the present invention also provides a controller, such as Fig. 20 As shown, the controller includes a processor 10 , a memory 20 and a display 30 . Fig. 20 Only some of the components of the controller are shown, but it should be understood that implementation of all of the components shown is not required, and more or fewer components may be implemented instead.

[0183] In some embodiments, the memory 20 may be an internal storage unit of the controller, such as a hard disk or memory of the controller. In other embodiments, the memory 20 may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the memory 20 may also include both an internal storage unit of the controller and an external storage device. The memory 20 is used to store application software and various types of data installed on the controller. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a target animal motion tracking program 40 based on a video sequence is stored on the memory 20, and the target animal motion tracking program 40 based on a video sequence can be executed by the processor 10, thereby realizing the target animal motion tracking method based on a video sequence in the present invention.

[0184] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the target animal motion tracking method based on the video sequence.

[0185] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the device and to display a visual user interface. The components 10-30 of the device communicate with each other via a system bus.

[0186] In one embodiment, when the processor 10 executes the target animal motion tracking program 40 based on the video sequence in the memory 20, the following steps are implemented:

[0187] Extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain enhanced third images; wherein all images in the third image set are the third images;

[0188] Performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box;

[0189] Inputting the enhanced third image and the first annotated file simultaneously into the optimized YOLOv4-Tiny model for training to obtain a parameter model;

[0190] After acquiring the fourth image captured by the calibrated binocular camera, preprocessing the fourth image to obtain a preprocessed fourth image, and inputting the preprocessed fourth image into the parameter model to obtain a detection frame of the target object and a confidence level of the target object;

[0191] After performing a pixel matching operation on two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, the depth of the pixels is calculated to obtain a fifth image; wherein the two images captured by the left and right cameras are both referred to as the fourth images;

[0192] The detection box is matched with the bounding box in the fifth image, and the successfully matched fifth image, the confidence level and the detection box are input into an improved Deep SORT algorithm model, and the improved Deep SORT algorithm model predicts and outputs motion behavior information of the target object.

[0193] The step of extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain the enhanced third image also includes:

[0194] The binocular camera is calibrated using a single-plane chessboard camera calibration method to obtain an internal and external parameter transformation matrix and a lens distortion coefficient;

[0195] The binocular camera is calibrated using the lens distortion coefficient.

[0196] The step of extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain an enhanced third image specifically includes:

[0197] Using the calibrated binocular camera to photograph the target object to obtain the second image set, and extracting the same number of the first images and the second images from the first image set and the second image set in a uniform random sampling manner to obtain the third image set;

[0198] All the third images in the third image set are enhanced by translation, rotation and noise addition to obtain enhanced third images.

[0199] The step of performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box specifically includes:

[0200] Classifying and frame processing the second image to obtain the second image with the boundary of the target object;

[0201] Using labelimg software, the boundary of the target object is marked with a rectangular box to obtain a first marking file with the boundary box of the target object.

[0202] The step of simultaneously inputting the enhanced third image and the first annotation file into the optimized YOLOv4-Tiny model for training to obtain a parameter model specifically includes:

[0203] After replacing the backbone feature extraction network of the original YOLOv4-Tiny model with the Mobilenetv3 structure, the number of layers of the Mobilenetv3 structure is increased, and the inverted residual structure is added to obtain the optimized YOLOv4-Tiny model;

[0204] The enhanced third image and the first annotation file are trained using the optimized YOLOv4-Tiny model to obtain the parameter model.

[0205] Among them, after acquiring the fourth image captured by the calibrated binocular camera, performing a preprocessing operation on the fourth image to obtain the preprocessed fourth image, and inputting the preprocessed fourth image into the parameter model, the step of obtaining the detection frame of the target object and the confidence of the target object specifically includes:

[0206] After acquiring the fourth image captured by the calibrated binocular camera, a Gaussian filter operation is performed on the fourth image using the open source OpenCV library to obtain a preprocessed fourth image;

[0207] The preprocessed fourth image is input into the parameter model to obtain the detection frame and the confidence level.

[0208] Among them, the step of performing pixel matching operation on the two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time and calculating the depth of the pixels to obtain the fifth image specifically includes:

[0209] After preprocessing the two preprocessed fourth images captured by the left and right cameras at the same time, pixel matching is performed on the two preprocessed fourth images according to the internal and external parameter transformation matrix;

[0210] The depth of each pixel is calculated according to the matching result to obtain all the fifth images; wherein the fifth image is a depth map.

[0211] The step of matching the detection box with the bounding box in the fifth image, and inputting the successfully matched fifth image, the confidence and the detection box into the improved Deep SORT algorithm model, wherein the improved DeepSORT algorithm model predicts and outputs the motion behavior information of the target object specifically includes:

[0212] The ShuffleNetv2 network is used to replace the ReID network in the original Deep SORT algorithm model to obtain the improved Deep SORT algorithm model;

[0213] Matching the detection frame with the bounding box in the fifth image, and creating a tracker in the improved Deep SORT algorithm model using the successfully matched fifth image and the detection frame;

[0214] After initializing the Kalman filter in the Deep SORT algorithm model, the Kalman filter is used to predict the state of the target object, and a prediction frame is obtained and saved in the tracker;

[0215] Filtering out the detection frames whose confidence is less than or equal to a preset threshold, and performing threshold matching and non-maximum suppression operations on the tracker and the filtered detection frames;

[0216] The trackers that have successfully matched the threshold value of the determined state are subjected to a cascade matching operation, and the Kalman filter is updated with the trackers that have successfully matched the cascade matching, so that the Kalman filter outputs the motion behavior information of the target object.

[0217] Furthermore, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a target animal motion tracking program based on a video sequence, wherein the target animal motion tracking program based on a video sequence, when executed by a processor, implements the steps of the target animal motion tracking method based on a video sequence as described above; since the steps of the target animal motion tracking method based on a video sequence are described in detail above, they will not be repeated here.

[0218] In summary, the present invention provides a method for tracking the motion of a target animal based on a video sequence and related equipment. The method for tracking the motion of a target animal based on a video sequence comprises the following steps: extracting a plurality of first images from a first image set and a plurality of second images from a second image set captured by a calibrated binocular camera, combining the plurality of first images and second images to obtain a third image set, performing enhancement processing on all third images in the third image set to obtain an enhanced third image; wherein all images in the third image set are the third images; performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box; and simultaneously inputting the enhanced third image and the first annotation file into an optimized YOLOv4 -Tiny model to obtain a parameter model; after collecting the fourth image captured by the calibrated binocular camera, the fourth image is preprocessed to obtain the preprocessed fourth image, and is input into the parameter model to obtain the detection frame of the target object and the confidence of the target object; after pixel matching operation is performed on the two preprocessed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, the depth of the pixel is calculated to obtain the fifth image; wherein, the two images captured by the left and right cameras are both called the fourth images; the detection frame is matched with the bounding box in the fifth image, and the matched fifth image, the confidence and the detection frame are input into the improved Deep SORT algorithm model, and the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object. By optimizing the YOLOv4-Tiny training model and inputting the depth map after the detection frame is successfully matched with the depth map into the improved Deep SORT algorithm model, the neural network model is made lightweight while ensuring the analysis accuracy.

[0219] It is understandable that those skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention, and all these changes or substitutions should fall within the protection scope of the claims attached to the present invention.

Claims

1. A method for tracking target animal motion based on video sequences, characterized in that: The method for tracking the motion of a target animal based on a video sequence comprises the following steps: Extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain enhanced third images; wherein all images in the third image set are the third images; Performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box; Inputting the enhanced third image and the first annotated file simultaneously into the optimized YOLOv4-Tiny model for training to obtain a parameter model; The step of simultaneously inputting the enhanced third image and the first annotation file into the optimized YOLOv4-Tiny model for training to obtain a parameter model specifically includes: After replacing the backbone feature extraction network of the original YOLOv4-Tiny model with the Mobilenetv3 structure, the number of layers of the Mobilenetv3 structure is increased, and the inverted residual structure is added to obtain the optimized YOLOv4-Tiny model; Using the optimized YOLOv4-Tiny model to train the enhanced third image and the first annotation file to obtain the parameter model; After acquiring the fourth image captured by the calibrated binocular camera, preprocessing the fourth image to obtain a preprocessed fourth image, and inputting the preprocessed fourth image into the parameter model to obtain a detection frame of the target object and a confidence level of the target object; After performing a pixel matching operation on two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, the depth of the pixels is calculated to obtain a fifth image; wherein the two images captured by the left and right cameras are both referred to as the fourth images; Matching the detection frame with the bounding box in the fifth image, and inputting the successfully matched fifth image, the confidence level and the detection frame into an improved Deep SORT algorithm model, wherein the improved Deep SORT algorithm model predicts and outputs motion behavior information of the target object; The step of matching the detection frame with the bounding box in the fifth image, and inputting the successfully matched fifth image, the confidence and the detection frame into the improved Deep SORT algorithm model, wherein the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object specifically includes: The ShuffleNetv2 network is used to replace the ReID network in the original Deep SORT algorithm model to obtain the improved Deep SORT algorithm model; Matching the detection frame with the bounding box in the fifth image, and creating a tracker in the improved Deep SORT algorithm model using the successfully matched fifth image and the detection frame; After initializing the Kalman filter in the improved Deep SORT algorithm model, the Kalman filter is used to predict the state of the target object, and a prediction frame is obtained and saved in the tracker; Filtering out the detection frames whose confidence is less than or equal to a preset threshold, and performing threshold matching and non-maximum suppression operations on the tracker and the filtered detection frames; The trackers that have successfully matched the threshold value of the determined state are subjected to a cascade matching operation, and the Kalman filter is updated with the trackers that have successfully matched the cascade matching, so that the Kalman filter outputs the motion behavior information of the target object.

2. The method for tracking target animal motion based on video sequence according to claim 1, characterized in that: Before the step of extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain an enhanced third image, the step further includes: The binocular camera is calibrated using a single-plane chessboard camera calibration method to obtain an internal and external parameter transformation matrix and a lens distortion coefficient; The binocular camera is calibrated using the lens distortion coefficient.

3. The method for tracking target animal motion based on video sequence according to claim 1, characterized in that: The steps of extracting a plurality of first images from the first image set and a plurality of second images from the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain an enhanced third image specifically include: Using the calibrated binocular camera to photograph the target object to obtain the second image set, and extracting the same number of the first images and the second images from the first image set and the second image set in a uniform random sampling manner to obtain the third image set; All the third images in the third image set are enhanced by translation, rotation and noise addition to obtain enhanced third images.

4. The method for tracking target animal motion based on video sequence according to claim 1, characterized in that: The step of performing data annotation processing on the second image to obtain a first annotation file with a target object boundary box specifically includes: Classifying and frame processing the second image to obtain the second image with the boundary of the target object; Using labelimg software, the boundary of the target object is marked with a rectangular box to obtain a first marking file with the boundary box of the target object.

5. The method for tracking target animal motion based on video sequence according to claim 2, characterized in that: The step of performing pixel matching operation on two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time and calculating the depth of the pixels to obtain the fifth image specifically includes: After preprocessing the two preprocessed fourth images captured by the left and right cameras at the same time, pixel matching is performed on the two preprocessed fourth images according to the internal and external parameter transformation matrix; The depth of each pixel is calculated according to the matching result to obtain all the fifth images; wherein the fifth image is a depth map.

6. A target animal motion tracking system based on video sequences, characterized in that: The target animal motion tracking system based on video sequence comprises: An image processing module, used for extracting a plurality of first images in the first image set and a plurality of second images in the second image set captured by the calibrated binocular camera, combining the plurality of first images and the second images to obtain a third image set, and performing enhancement processing on all third images in the third image set to obtain an enhanced third image; wherein all images in the third image set are the third images; A data annotation module, used to perform data annotation processing on the second image to obtain a first annotation file with a target object boundary box; A parameter model building module, used for simultaneously inputting the enhanced third image and the first annotation file into the optimized YOLOv4-Tiny model for training to obtain a parameter model; The step of simultaneously inputting the enhanced third image and the first annotation file into the optimized YOLOv4-Tiny model for training to obtain a parameter model specifically includes: After replacing the backbone feature extraction network of the original YOLOv4-Tiny model with the Mobilenetv3 structure, the number of layers of the Mobilenetv3 structure is increased, and the inverted residual structure is added to obtain the optimized YOLOv4-Tiny model; Using the optimized YOLOv4-Tiny model to train the enhanced third image and the first annotation file to obtain the parameter model; A training result prediction module, used for collecting a fourth image captured by the calibrated binocular camera, performing a preprocessing operation on the fourth image to obtain a preprocessed fourth image, and inputting the preprocessed fourth image into the parameter model to obtain a detection frame of the target object and a confidence level of the target object; A depth map acquisition module, used for performing a pixel matching operation on two pre-processed fourth images captured by the left and right cameras of the calibrated binocular camera at the same time, and calculating the depth of the pixels to obtain a fifth image; wherein the two images captured by the left and right cameras are both referred to as the fourth images; A motion behavior information output module, used for matching the detection frame with the bounding box in the fifth image, and inputting the successfully matched fifth image, the confidence level and the detection frame into an improved Deep SORT algorithm model, wherein the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object; The step of matching the detection frame with the bounding box in the fifth image, and inputting the successfully matched fifth image, the confidence and the detection frame into the improved Deep SORT algorithm model, wherein the improved Deep SORT algorithm model predicts and outputs the motion behavior information of the target object specifically includes: The ShuffleNetv2 network is used to replace the ReID network in the original Deep SORT algorithm model to obtain the improved Deep SORT algorithm model; Matching the detection frame with the bounding box in the fifth image, and creating a tracker in the improved Deep SORT algorithm model using the successfully matched fifth image and the detection frame; After initializing the Kalman filter in the improved Deep SORT algorithm model, the Kalman filter is used to predict the state of the target object, and a prediction frame is obtained and saved in the tracker; Filtering out the detection frames whose confidence is less than or equal to a preset threshold, and performing threshold matching and non-maximum suppression operations on the tracker and the filtered detection frames; The trackers that have successfully matched the threshold value of the determined state are subjected to a cascade matching operation, and the Kalman filter is updated with the trackers that have successfully matched the cascade matching, so that the Kalman filter outputs the motion behavior information of the target object.

7. A controller, characterized in that: The controller includes: a memory, a processor, and a video sequence-based target animal motion tracking program stored in the memory and executable on the processor. When the video sequence-based target animal motion tracking program is executed by the processor, the steps of the video sequence-based target animal motion tracking method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a target animal motion tracking program based on a video sequence, and when the target animal motion tracking program based on a video sequence is executed by a processor, the steps of the target animal motion tracking method based on a video sequence as described in any one of claims 1-5 are implemented.