Visual intelligent perception technology and target visual tracking technology

Abstract

According to the visual intelligent perception technology and the target visual tracking technology, video monitoring for a task area is a behavior main body for constructing an early warning system as a group emergency, and individual units such as people and vehicles are the most important analysis objects. The analysis prerequisite condition is that detection identification, attribute extraction and positioning tracking of the main units need to be completed. In order to more accurately analyze a target behavior, the target behavior is distinguished into a motion behavior and a micro behavior according to whether the target behavior is along with the change of a motion track or not in the occurrence process of the target behavior. And clues and bases can be provided for subsequent event analysis and early warning for people regardless of abnormities of motion behaviors and micro behaviors. Therefore, after the target detection is completed, the item synchronously carries out the analysis and recognition of the motion behavior and the micro behavior of each main target, and completes the recognition of a specific abnormal behavior through training a targeted deep intelligent model.

Description

Technical Field

[0001] The present application relates to a computer vision-based target detection technology and a target tracking technology, and specifically to the use of target detection and target tracking technology to monitor camera images in real time, and to discover abnormal states of targets at the first time, and to be able to issue warnings and provide useful information in the fastest way, and to efficiently assist monitoring personnel in obtaining accurate information and handling emergencies. Background Art

[0002] In order to use artificial intelligence video analysis technology to monitor and warn of mass emergencies that may occur in the mission area, in theory, a large amount of video data of emergencies in the mission area is needed as samples for model training. However, the reality is that each emergency event is unique and unpredictable, and the video image data of mass emergencies collected in the early stage is not enough to ensure the training of high-precision video analysis intelligent models.

[0003] Based on this, this project intends to mainly study the behavioral status of monitoring targets in uncertain environments. It is necessary to accurately capture the monitoring targets using target detection technology and locate the monitoring targets using tracking technology: walking trajectory, walking speed, social distance, walking direction, and use this information to analyze the behavioral status of the monitoring targets. Summary of the invention

[0004] The purpose of this application is to propose a computer vision target detection technology and a target tracking technology, which can realize real-time monitoring of the monitoring target in the image and analyze the behavior status of the target person.

[0005] In order to achieve the above objectives, this application combines computer vision technology to propose computer vision-based target detection technology and target tracking technology. First, use labelimg to annotate the image, and divide the annotated data set script into a training set, a validation set, and a test set. Figure 1 .

[0006] The technical algorithm calculates the behavior state of the monitored target. Therefore, before real-time monitoring, it is necessary to configure the data and matching files required for training. This will improve our robustness and accuracy during subsequent reasoning. In this application, the target detection method uses deep learning technology based on deep learning neural network (YOLOv5). The target detection technology based on deep learning algorithm not only has good performance but also strong robustness.

[0007] The architecture of the YOLO model algorithm used in the target detection algorithm in this application method is described in detail. The architecture of its deep learning model is as follows Figure 2As shown in the figure, the network structure of YOLO v5 is divided into three parts: backbone network, neck structure, and head structure. Figure 2 .

[0008] 1. Backbone network

[0009] The backbone network refers to the network used to extract image features. Its main function is to convert the original input image into a multi-layer feature map for subsequent target detection tasks. In Yolov5, CSPDarknet53 or ResNet backbone networks are used. Both networks are relatively lightweight and can minimize the amount of computation and memory usage while ensuring high detection accuracy.

[0010] The main structures in Backbone include Conv module, C3 module, and SPPF module.

[0011] The Conv module is a commonly used basic module in convolutional neural networks. It extracts local spatial information through convolution operations, normalizes the eigenvalue distribution through the BN layer, and finally introduces nonlinear transformation capabilities through the activation function, thereby realizing the conversion and extraction of input features.

[0012] The C3 module improves the feature extraction capability by increasing the depth and receptive field of the network. This is very important for computer vision tasks such as target detection, because these tasks require accurate recognition and positioning of objects, which requires good feature extraction capabilities.

[0013] The SPP module is a pooling module that is usually used in convolutional neural networks. It aims to achieve spatial invariance and position invariance of input data in order to improve the recognition ability of neural networks. The main idea is to apply receptive fields of different sizes to the same image so as to capture feature information of different scales. In the SPP module, the input feature map is first pooled with different sizes to obtain a set of feature maps of different sizes. These feature maps are then connected together and reduced in dimension through a fully connected layer to finally obtain a feature vector of fixed size.

[0014] 2. Neck feature pyramid

[0015] Since the size and position of objects in the image are uncertain, a mechanism is needed to handle targets of different scales and sizes. Feature pyramid is a technique for handling multi-scale target detection, which can be achieved by adding feature layers of different scales to the backbone network. In Yolov5, the FPN (Feature Pyramid Network) feature pyramid structure is used to fuse feature maps of different levels through upsampling and downsampling operations to generate a multi-scale feature pyramid. The top-down part mainly achieves the fusion of features at different levels by upsampling and fusing with coarser-grained feature maps, while the bottom-up part fuses feature maps from different levels by using a convolutional layer.

[0016] In the target detection algorithm, the Neck module is usually used to combine feature maps of different levels to generate feature maps with multi-scale information to improve the accuracy of target detection. In YOLOv5, a feature fusion module called PANet is used as the Neck module.

[0017] Specifically, the top-down part is to achieve the fusion of features at different levels by upsampling and fusing with coarser-grained feature maps, which is mainly divided into the following steps:

[0018] ●Upsample the last layer of feature maps to obtain a finer feature map;

[0019] ●Fuse the upsampled feature map with the feature map of the previous layer to obtain richer feature expression;

[0020] ●Repeat the above two steps until you reach the highest level.

[0021] The bottom-up part mainly uses a convolutional layer to fuse feature maps from different levels, which is mainly divided into the following steps:

[0022] ●Convolve the bottom-level feature map to obtain richer feature expressions;

[0023] ●Fuse the convolutional feature map with the feature map of the previous layer to obtain richer feature expression;

[0024] ●Repeat the above two steps until you reach the highest level.

[0025] Finally, the feature maps of the top-down part and the bottom-up part are fused to obtain the final feature map for target detection.

[0026] 3.Head target detection head

[0027] The object detection head is used to detect objects on the feature pyramid. It includes some convolutional layers, pooling layers, and fully connected layers. In the YOLOv5 model, the detection head module is mainly responsible for multi-scale object detection on the feature map extracted by the backbone network. This module mainly consists of three parts. In addition, Yolov5 also uses some techniques to further improve the detection accuracy, such as GIoU loss, Mish activation function, and multi-scale training.

[0028] Anchors: used to define target boxes of different sizes and aspect ratios. They are usually obtained by clustering the target boxes of the training set using K-means clustering. They can be calculated before model training and stored in the model to generate detection boxes during prediction.

[0029] Classification: It is used to classify each detection box to determine whether it is a target object. It usually uses a fully connected layer plus a Softmax function to classify features.

[0030] Regression: It is used to regress each detection box to obtain its position and size. It usually regresses the features in the form of a fully connected layer.

[0031] The detection layer of YOLOv5 consists of several important components, including:

[0032] Anchors:

[0033] Anchor boxes are a set of predefined bounding boxes used to generate candidate boxes on feature maps.

[0034] YOLOv5 adapts to objects of different sizes by defining anchor boxes of different scales and sizes in advance.

[0035] Convolutional Layers:

[0036] The detection layer of YOLOv5 contains a series of convolutional layers to process feature maps and extract features.

[0037] These convolutional layers can be adapted to different detection tasks by adjusting the number of channels and kernel size.

[0038] Prediction Layers:

[0039] Each prediction layer is responsible for predicting a set of bounding boxes and categories.

[0040] Each prediction layer usually consists of a convolutional layer and an output layer.

[0041] The number of channels and shape of the output layer determine the number of predicted bounding boxes and the number of categories.

[0042] Non-Maximum Suppression (NMS):

[0043] In the output bounding box, the non-maximum suppression algorithm is used to suppress overlapping bounding boxes and only retain the most representative bounding boxes.

[0044] The architecture of the ByteTrack model algorithm used in the target tracking algorithm in the method of this application is described in detail. ByteTrack is a tracking method based on the tracking-by-detection paradigm. A simple and efficient data association method BYTE is proposed. The biggest difference between it and the previous tracking algorithm is that it does not simply remove the low-score detection results, Assiciating Every Detection Box. By using the similarity between the detection box and the tracking trajectory, while retaining the high-score detection results, the background is removed from the low-score detection results, and the real objects (difficult samples such as occlusion and blur) are excavated, thereby reducing missed detections and improving the coherence of the trajectory. All indicators have breakthroughs. ByteTrack's improvement in occlusion is very obvious. However, it should be noted that since ByteTrack does not use appearance features for matching, the tracking effect is very dependent on the detection effect. That is to say, if the detector effect is very good, the tracking will also achieve good results, but if the detection effect is not good, it will seriously affect the tracking effect.

[0045] ByteTrack is a core technology based on BYTE. It is flexible and versatile, allowing users to select and integrate their own target detection algorithms according to their needs. By using the output of the custom detection algorithm as input, ByteTrack implements the target tracking function. In the field of engineering applications, ByteTrack is widely used for its simplicity and efficiency.

[0046] The specific process of the method is as follows:

[0047] ● According to the detection frame scores, the detection frames are divided into high-score frames and low-score frames, and processed separately

[0048] ●First use of high-resolution frame and previous tracking track matching

[0049] ● The second time, the low-scoring frame is used for matching with the tracking tracks that did not match the high-scoring frame in the first time (such as objects that are severely occluded in the current frame, causing the score to drop).

[0050] ●For the detection box with a high enough score that does not match the tracking track, we will create a new tracking track. For the tracking track that does not match the detection box, we will keep 30 frames and match it when it appears again.

[0051] The working principle of BYTE can be understood as follows: occlusion is often accompanied by a slow decrease in detection scores from high to low: the occluded object was a visible object before being occluded. The higher the detection score, the easier it is to establish a trajectory. When the object is occluded, the occluded object can be extracted from the low-score frame by the overlap between the detection frame and the trajectory, thus maintaining the continuity of the trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The format of the training dataset is

[0053] Figure 2 YOLOv5 data structure and network flow chart for target detection

[0054] Figure 3 Data structure and network flow chart for the tracking algorithm ByteTrack

[0055] Figure 4 The configuration file format is yaml Figure 5 Detection effect for detect.py Specific implementation methods

[0057] In order to better describe the computer vision-based target detection technology and target tracking technology, the specific implementation of the present application is given below.

[0058] (1) Before using train., you need to prepare your own dataset. The dataset format is as follows: Figure 1 ,

[0059] There are two folders under the dataset file, images and labels. The images folder is the train folder, which is the training image folder, and the val folder is the validation image folder. The labels folder is the train folder, which is the training label folder, and the val folder is the validation label folder.

[0060] (2) yaml is the configuration file used for model training on a dataset. Its function is to provide appropriate parameter settings for the dataset for model training, such as Figure 4 .

[0061] train is the storage path of the training images, val is the storage path of the verification images, test is the specified image path of the test images, and it can also be unspecified. The number of types of objects to be detected in the nc dataset, and names specifies the category names of the objects to be detected in the dataset.

[0062] (3) Prepare pre-trained models and configure training model files

[0063] The purpose of a pre-trained model is to provide a detection model that has been pre-trained on a large-scale dataset, which can be used to quickly develop programs for target detection applications. By training on a large-scale dataset, the pre-trained model can effectively learn common feature representations that can be used to identify the categories of various different objects.

[0064] In the train.py file, the def parse_opt() function records the relevant command line parameters, and performs necessary checks and processing on some command line parameters (such as path checks, value range checks, etc.) to ensure that the program can run normally. Next, we will introduce several necessary and commonly used parameters.

[0065] ●weights is the parameter name of the model weights, which generally refers to the pre-trained weight file that needs to be loaded into the model

[0066] ●cfg is the name of the configuration file parameter of the model structure (short for config). This parameter allows the user to load the specified configuration file to initialize the structure of the current model

[0067] ●data is the parameter name of the dataset configuration file. This parameter allows the user to load the specified dataset configuration file in order to initialize the dataset and perform model training or inference.

[0068] ●hyp is the name of the hyperparameter configuration file parameter. This parameter allows the user to load the specified hyperparameter configuration file to initialize the hyperparameters and perform model training.

[0069] ●Epochs refers to the number of rounds of model training. This parameter allows users to set the number of rounds that the model needs to be trained in order to achieve better model performance

[0070] ●Batch-size refers to the number of images input to the model in each batch. This parameter allows users to set the number of images input per batch during model training in order to achieve better training results.

[0071] ●imgsz is the size name of the input image. This parameter allows users to set the size of the input image to suit different application scenarios and hardware configurations.

[0072] ●Resume refers to the operation of resuming model training from the interrupted position after model training is interrupted. This operation usually requires the use of the file saved in the model training status

[0073] ●Device is a class in the PyTorch library that manages the data storage location of the device. When using PyTorch for deep learning calculations, we can use the device class to specify the device on which the calculation is run. Common devices include CPU and GPU.

[0074] ●Optimizer is an algorithm that updates model parameters. Usually, our goal is to train a neural network model to achieve the best performance on a given training data set. The optimizer can update the model parameters based on the training data and the model's loss function to make the model's predictions as close to the true value as possible.

[0075] The project parameter is usually used to specify the project name and path for saving the output results during the training process. When training a neural network, we need to save the training data set, model, log and other information to a specified directory to facilitate subsequent debugging and tracking.

[0076] (4) Operation results

[0077] ●Environment information, including version number, GPU / CPU device information, etc.

[0078] ●Model summary information, including network structure, number of parameters, number of gradients, amount of calculation, etc.

[0079] ●Configuration information during training, including data set, hyperparameters, optimizer, learning rate, etc.

[0080] ● Training system information, including the current round number, current batch number, average loss, prediction accuracy, etc.

[0081] ●Save model parameters, intermediate results, log files, etc.

[0082] ●Show the changing trends of various indicators during training in TensorBoard

[0083] (5) Model Reasoning

[0084] Model inference detect.py file is a script for using the trained model to perform target detection tasks. It can receive a single image, video or camera, and input the detection result image or video. detect.py provides a simple and easy-to-use interface, allowing developers to use the trained model to quickly perform target detection tasks, which is convenient for practical applications. The use effect is as follows Figure 5 .

[0085] The detect.py script contains the following main functions:

[0086] ● Load the trained model weights and create the corresponding network structure

[0087] ●Receive user-defined input (single image, video stream, or camera), process and pre-process the input data

[0088] ●Use the model to detect the target and get the detection results

[0089] ●Submit the target detection results to the tracker to obtain the content that needs to be tracked

[0090] ●Visualize the test results and output them to the screen or save them to a specified file

[0091] (6) Target tracking detection configuration and use

[0092] Enter the detect.py file in the main directory. The ByteTrack_opt() function records the relevant command line parameters. Some command line parameters are checked and processed as necessary (such as drawing, value range check, etc.) to ensure that the program can run normally. Next, we will introduce several necessary and commonly used parameters.

[0093] ●track_thresh is a hyperparameter in the ByteTrack algorithm, which is used to specify the target matching threshold. Specifically, when the feature distance between two targets is less than track_thresh, they are considered to be the same target and are tracked; when the feature distance between two targets is greater than track_thresh, they are considered to be two different targets and the detection and tracking process is restarted.

[0094] ●track_buffer is a hyperparameter in the ByteTrack algorithm, which is used to specify the size of the target buffer. Specifically, Track_buffer indicates how many consecutive frames after which the target is not detected before the target is marked as lost (i.e., does not exist in the current frame)

[0095] ●Match_thersh is a hyperparameter in the ByteTarch algorithm, which is used to specify the threshold for target matching. Specifically, when the feature distance between two targets is less than match_thersh, they are considered to be the same target and are matched and tracked; otherwise, they are considered to be different targets and need to be re-detected and relocated.

[0096] ●Aspect_ratio_thersh is a hyperparameter in the ByteTrack algorithm, which is used to specify the threshold of the length and width of the target. Specifically, when the aspect ratio (i.e. the ratio of the long side to the wide side) of the detected target is less than or greater than aspect_ratio_thersh, the target is filtered out and not tracked.

[0097] ●Min_box_area is a hyperparameter in the ByteTrack algorithm, which is used to specify the area threshold of the minimum external matrix of the detected target. Specifically, when the area of ​​the external matrix of the detected target is smaller than min_box_area, the target is filtered out and not tracked or matched.

Claims

1. Visual intelligent perception technology and target visual tracking technology, characterized by: Identify the targets in the monitoring system, and continuously track and analyze the target's behavior status to ensure that abnormal behaviors are handled in the first place. The specific steps are as follows: Step 1) Prepare training images and use labeimg for data annotation; Step 2) Configure the yaml file and train.py file to prepare the pre-training weights; Step 3) Use the train.py file to train, save and analyze the trained file; Step 4) Prepare test data and configure the detect.py detection file; Step 5) Adjust target tracking parameters; Step 6) Use the trained weight file to detect the target.

2. The computer vision-based target detection and target tracking technology according to claim 1 (characterized in that Step 1) Use labelimg annotation tool to describe the data, and each label is in yolo format.

3. The computer vision-based target detection and target tracking technology according to claim 1 (characterized in that Step 2) Divide the labeled data files into training set, test set, and validation set, and record the data address, number of categories, and category name in the yaml file, and configure the train.py file according to the requirements.

4. The computer vision-based target detection and target tracking technology according to claim 1 (characterized in that Step 3), use train.py to train and analyze the map, loss, and ap values ​​after training to see if they meet the requirements.

5. The computer vision-based target detection and target tracking technology according to claim 1 (characterized in that Step 4) Prepare the images to be detected and modify the detect.py detection file as required.

6. The computer vision-based target detection and target tracking technology according to claim 1 (characterized in that Step 5) Adjust the target tracking parameters. Parameters that need to be adjusted: tracking confidence threshold, number of consecutive tracking frames, tracking matching threshold, etc.

7. The computer vision-based target detection and target tracking technology according to claim 1 (characterized in that Step 6) Prepare the trained weight file to perform inference detection on the detection image.

Images