Target recognition tracking method and device, electronic equipment and storage medium

Abstract

The application provides a target identification and tracking method and device, electronic equipment and a storage medium. The method comprises the following steps: constructing a first neural network model for training target identification based on a RetinaNet network and constructing a second neural network model for training target tracking based on a deepSORT algorithm; obtaining a video stream of a preset monitoring area, and inputting a picture frame of the preprocessed video stream into the trained first neural network model to obtain a target detection result of each target in the picture frame; and inputting the target detection result of at least one target into the trained second neural network model for position prediction to obtain trajectory information corresponding to the target. The application can accurately identify targets and meet the demand for target trajectory tracking.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a target recognition and tracking method, apparatus, electronic device, and storage medium. Background Technology

[0002] For example, in the shipbuilding industry, monitoring of special working environments on board is an important part of daily ship operation and maintenance and personnel protection. Confirming the behavior and movement of personnel in various compartments such as the bridge and engine room is also crucial. Therefore, it is necessary to monitor and record personnel movement and conduct behavioral identification by identifying unauthorized personnel entering sensitive work areas.

[0003] In the field of multi-target tracking, commonly used algorithms employ manually designed features, such as optical flow, particle filtering, and mean-shift filtering. However, these algorithms are affected by multiple factors, including scene noise, target movement speed, and frame rate selection. Therefore, the target features in traditional multi-target tracking algorithms require expert parameter tuning, resulting in significant algorithmic uncertainty, making it difficult to accurately identify targets and meet the needs of target movement tracking and monitoring. Summary of the Invention

[0004] This invention provides a target identification and tracking method, apparatus, electronic device, and storage medium to solve the problems in the prior art of difficulty in accurately identifying targets and inability to meet the needs of target movement tracking and monitoring.

[0005] In a first aspect, the present invention provides a target identification and tracking method, the method comprising:

[0006] A first neural network model for training target recognition was constructed based on the RetinaNet network, and a second neural network model for training target tracking was constructed based on the deepSORT algorithm.

[0007] The video stream of the preset monitoring area is acquired, and the pre-processed video stream frames are input into the trained first neural network model to obtain the target detection result of each target in the frame.

[0008] The target detection results of at least one target are input into a trained second neural network model to predict the location and obtain the trajectory information corresponding to the target.

[0009] In one embodiment of the present invention, the step of constructing a first neural network model for training target recognition based on a RetinaNet network includes:

[0010] A video processing module and a detection algorithm module are constructed using transfer learning based on the RetinaNet network to build the training environment for the first neural network model;

[0011] The training environment includes extracting image features using the ResNet and FPN networks in the RetinaNet network, searching for bounding boxes using anchor boxes, calling the Focal loss function in the class subnet to predict the class, and using the box subnet to predict the bounding box coordinates and size.

[0012] In one embodiment of the present invention, the steps of performing transfer learning using the RetinaNet network include:

[0013] Modify the parameters of the configuration file of the preset model to adapt to the training requirements of the RetinaNet network. The parameters include one or more combinations of the number of training target categories, the resized training images, the classification model, the training data path, the evaluation data path, and the label index path.

[0014] Retain the feature extraction layer and bounding box regression prediction layer of the RetinaNet network and freeze the object classification prediction layer that is not used in the RetinaNet network;

[0015] The training mode of the first neural network model is set and the training parameters are configured for model training. The training parameters include one or more combinations of the number of samples in one training session, the number of training batches, the learning rate, and the optimization method.

[0016] During training, tracking parameters are set to select the optimal model as the first neural network model.

[0017] In one embodiment of the present invention, the step of constructing a second neural network model for training target tracking based on the deepSORT algorithm includes:

[0018] The deepSORT tracker is trained using the cosine_metric_learning training repository of the deepSORT algorithm, and the training results are exported using the TensorFlow framework. The exported training results are frozen pb models.

[0019] The model from which the training results are derived is used as the second neural network model.

[0020] In one embodiment of the present invention, the steps of acquiring a video stream of a preset monitoring area and inputting the preprocessed video stream frames into a trained first neural network model to obtain the target detection result of each target in the frame include:

[0021] By connecting to a camera device, a video stream of a preset monitoring area is obtained. The parameters of the video stream include the number of frames per second, the width and height of each frame, and one or more combinations of pixels.

[0022] Initialize the tracker, set relevant parameters and calculate the cosine distance metric, set the maximum cosine distance between targets in two frames of images using the tracker, and control the calculation of features of adjacent targets;

[0023] The video stream is preprocessed frame by frame, each frame of the image is changed to the size processed by the first neural network model, and the parameters are adjusted according to different scenarios.

[0024] The frame is input into the first neural network model to obtain the target detection result of the frame.

[0025] In one embodiment of the present invention, the step of inputting the target detection result of at least one target into a trained second neural network model for position prediction to obtain the trajectory information corresponding to the target includes:

[0026] Create a corresponding track detection box based on the target detection results;

[0027] The second neural network model is used to perform secondary classification on the target detection results, and the target features, target box coordinates, target box categories and confidence scores extracted from the target detection results are converted into the data format input to the deepSORT tracker.

[0028] The location of the created track detection box is predicted based on the trajectory detected in the previous frames of the deepSORT tracker.

[0029] In one embodiment of the present invention, the step of predicting the position of the track detection box based on the trajectory detected in the previous frames of the deepSORT tracker includes:

[0030] The position of the track detection box at time t is predicted based on the position of the created track detection box at time t-1.

[0031] Based on the detected position at time t, update the positions of other track detection boxes associated with it to obtain the trajectory information corresponding to the target.

[0032] In one embodiment of the present invention, the step of predicting the position of the track detection box at time t based on the position of the created track detection box at time t-1 includes:

[0033] Determine the formulas for predicting the position of the track detection box and the covariance formula;

[0034] The formula for predicting the position of the track detection box is as follows:

[0035] x′=Fx;

[0036] x represents the mean value of the target location information corresponding to the track detection box at time t-1, which is determined by the center position coordinates (x, y) of the target bounding box. x ,x y It consists of aspect ratio r, height h, and velocity change values ​​corresponding to each track detection box, x′ represents the mean value of the target position information corresponding to the track detection box at time t, and F represents the state transition matrix;

[0037] The formula for the covariance of the predicted location of the detection box in this track is as follows:

[0038] P′=FPF T +Q;

[0039] P represents the covariance matrix of the track detection box at time t-1, Q represents the noise matrix of the system, and P′ represents the covariance matrix of the track detection box at time t.

[0040] In one embodiment of the present invention, the step of correcting the positions of other track detection boxes associated with the detected position at time t to obtain the trajectory information corresponding to the target includes:

[0041] The error between the target detection result and the mean value of the track detection box is calculated using the following formula:

[0042] y = z - Hx′;

[0043] Where z represents the mean vector of the track detection box, H represents the measurement matrix, and y represents the error value;

[0044] The mean vector x and covariance matrix P of the track detection box are updated using the following formula:

[0045] S=HP′H T +R;

[0046] x=x′+P′H T S -1 y;

[0047] P=(IP′H T S -1 H)P';

[0048] Where S represents the intermediate variable, R represents the noise matrix of the deepSORT tracker, and I represents the identity matrix;

[0049] Based on the updated track detection box, perform concatenated matching between the current track detection box and its associated track detection boxes;

[0050] Based on the results of the cascaded matching, the coordinate frame position of the target is output as the trajectory information of the target.

[0051] Secondly, the present invention also provides a target recognition and tracking device, the device comprising:

[0052] The model building module is used to build a first neural network model for training target recognition based on the RetinaNet network and a second neural network model for training target tracking based on the deepSORT algorithm.

[0053] The target detection module is used to acquire the video stream of the preset monitoring area and input the pre-processed video stream frames into the trained first neural network model to obtain the target detection result of each target in the frame.

[0054] The target prediction module is used to input the target detection results of at least one target into a trained second neural network model to predict the position and obtain the trajectory information corresponding to the target.

[0055] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the target recognition and tracking methods described above.

[0056] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the target recognition and tracking methods described above.

[0057] The target recognition and tracking method, device, electronic device, and storage medium provided by the present invention use a first neural network model based on the RetinaNet network to perform target recognition on the input video stream, and then use a second neural network model based on the deepSORT algorithm to track the target identified by the first neural network model, which can accurately identify the target and meet the requirements of target trajectory tracking. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating the target identification and tracking method provided by the present invention;

[0060] Figure 2 This is a schematic diagram of the process for constructing the first neural network model provided by the present invention;

[0061] Figure 3 This is a schematic diagram of the process for constructing the second neural network model provided by the present invention;

[0062] Figure 4 This is a schematic diagram of the detection process of the first neural network model provided by the present invention;

[0063] Figure 5 This is a schematic diagram of the prediction process of the second neural network model provided by the present invention;

[0064] Figure 6 This is a schematic diagram of the target recognition and tracking device provided by the present invention;

[0065] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0068] The technical terms involved in this invention are described below:

[0069] Multiple Object Tracking (MOT) is mainly about finding moving objects in a sequence of images and identifying moving targets in different frames, given a specific and accurate ID. However, these targets can be arbitrary, such as pedestrians, vehicles, and various animals.

[0070] Commonly used algorithms for multi-target tracking include SORT (Simple Online And Realtime Tracking) and deepSORT. The core of the SORT algorithm is the Kalman filter and the Hungarian matching algorithm. DeepSORT adds a discrimination network (Deep Association Metric), matching cascade, and confirmation of new trajectories to the SORT algorithm.

[0071] In the industrial field, a common approach is to combine deep learning target recognition algorithms with deepSORT tracking algorithms. Among these, YOLO, SSD, and RetinaNet networks are frequently used in target recognition algorithms.

[0072] To address the problems of inaccurate target identification and failure to meet the needs of target movement tracking and monitoring in existing technologies, the present invention provides a target identification and tracking method, device, electronic device, and storage medium. This method utilizes a first neural network model built on the RetinaNet network to identify targets in the input video stream. The targets identified by the first neural network model are then tracked using a second neural network model built on the deepSORT algorithm. This approach enables accurate target identification and meets the requirements for target trajectory tracking.

[0073] The target recognition and tracking method provided by this invention is a technical solution for tracking and monitoring the movement of people in a monitored area based on deep learning. It can be further combined with other recognition algorithms and applied to the analysis of different specific events, especially for tracking and recording the intrusion and behavior of non-staff members in workplaces with a clear range of activity.

[0074] The following is combined Figures 1-7 The present invention describes a target identification and tracking method, apparatus, electronic device, and storage medium.

[0075] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the target recognition and tracking method provided by the present invention. A target recognition and tracking method, the method comprising:

[0076] Step 110: Construct a first neural network model for training target recognition based on the RetinaNet network and a second neural network model for training target tracking based on the deepSORT algorithm.

[0077] Step 120: Obtain the video stream of the preset monitoring area, and input the pre-processed video stream frames into the trained first neural network model to obtain the target detection result of each target in the frame.

[0078] Step 130: Input the target detection result of at least one target into the trained second neural network model to perform position prediction and obtain the trajectory information corresponding to the target.

[0079] It should be noted that the RetinaNet network used in this invention introduces a focal loss function during the model training process, which can effectively solve the problem of imbalance between positive and negative samples in practical applications (e.g., there are many legitimate employee samples and few illegal intruder samples). Furthermore, this invention selects layers on the RetinaNet network and constructs a first neural network model suitable for detection, which is then converted into the data format used by the deepSORT algorithm in the second neural network model for updating and applying the track (detection box) and tracker.

[0080] The following is a detailed description of steps 110 to 130 above.

[0081] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the process for constructing the first neural network model provided by the present invention. For example, in step 110 above, the step of constructing the first neural network model for training object recognition based on the RetinaNet network includes:

[0082] Step 111: Utilize transfer learning based on the RetinaNet network to construct a video processing module and a detection algorithm module, and build the training environment for the first neural network model.

[0083] The training environment includes extracting image features using the ResNet and FPN networks in the RetinaNet network, searching for bounding boxes using anchor boxes, predicting the class by calling the Focal loss function in the class subnet, and predicting the bounding box coordinates and size using the box subnet.

[0084] This step involves training several different detector models, determined by the monitoring objectives, hardware requirements, and outcome expectations for different regions. Various metrics of the different models are matched to the specific needs of the implementation project, such as hardware computing power, the pixel clarity of the monitoring cameras, and the processing frame rate to match the model's processing speed.

[0085] For example, the backbone of the RetinaNet network is a ResNet network. ResNet uses 18 to 152 layers, depending on the specific application. Its function is to extract information from images for use by subsequent networks. Although the ResNet network already has the ability to extract image features, to better utilize the backbone to extract more complex features, this invention also configures an FPN network to extract image features on top of the ResNet network.

[0086] The RetinaNet network used in this invention is derived from single-object detection models (such as SSD or YOLO) through two improvements. Unlike other object detection models that suffer from severe imbalance between positive and negative sample regions during network training, making the object detection loss function susceptible to the influence of a large number of negative samples, RetinaNet improves upon the traditional cross-entropy (CE) loss function by employing Focal Loss. Focal Loss adjusts the weights for positive, negative, and difficult samples simultaneously by selecting smaller values ​​to reduce the weight of negative samples (those that are more numerous). By reducing the weight of easily distinguishable samples through the Focal Loss function, the model can focus more on learning difficult samples during training, thus improving the imbalance problem.

[0087] Specifically, the steps for transfer learning using the RetinaNet network described above include:

[0088] Step 1111: Modify the parameters of the preset model's configuration file to adapt to the training requirements of the RetinaNet network.

[0089] The parameters include, but are not limited to: the number of training target categories, the resized training images, changes to the classification model, the training data path, the evaluation data path, and the label index path. Unused batches of standard layers are frozen on the RetinaNet network.

[0090] It should be noted that the preset model mentioned in this invention refers to an existing model based on the RetinaNet network, and the first neural network model of this invention is obtained by modifying the configuration file of an existing model.

[0091] Step 1112: Retain the feature extraction layer and bounding box regression prediction layer of the RetinaNet network and freeze the unused object classification prediction layer in the RetinaNet network.

[0092] For example, the first neural network model is built using the build tools in the Object Recognition API. The model configuration file path and other parameters are set in the build function for initial model building. Weights generated by checkpoints in the pre-trained model are selectively recovered. The ultimate goal of building the model is to use weights from a portion of the pre-trained model instead of all layers in the RetinaNet network. During model building, the feature extraction layers and bounding box regression prediction layers in the RetinaNet network are retained, while the unused object classification prediction layers are frozen. For the few layers that need to be retained from the RetinaNet network structure, weights are selectively recovered from checkpoints in the pre-trained model.

[0093] To preserve network layers, define checkpoints. Isolate the network layers you want to retain from the initially built model, and use these checkpoints to selectively restore the weights of those specified layers.

[0094] First, a temporary checkpoint for bounding box prediction is defined using "tf.train.checkpoint". This function maintains two layers: the base tower layer and the boxprediction head.

[0095] Next, a temporary model checkpoint is defined using "tf.train.checkpoint". This function preserves the feature region layers from the initial model construction and specifies the bounding box prediction layers as the newly defined temporary bounding box prediction checkpoints.

[0096] Finally, the checkpoints path in the pre-downloaded model is specified through the defined temporary model checkpoints structure, and the layer weights in the checkpoints are restored using the "restore" function.

[0097] Step 1113: Set the training mode of the first neural network model and configure the training parameters for model training.

[0098] The training parameters include, but are not limited to: the number of samples in one training session, the number of training batches, the learning rate, and the optimization method.

[0099] For example, hyperparameters in the model training loop are set and "eager mode" is used. Tensorflow has two modes, "Graph mode" and "eager mode". Since this invention does not involve parallel computing between multiple GPUs, eager mode can be selected for configuration. Parameters set in the model training loop include, but are not limited to: the number of samples per training session, the number of training batches, the learning rate, and the optimization method.

[0100] By building upon the RetinaNet network and using transfer learning techniques on a pre-trained model, only the prediction layer of the last layer needs to be trained. Examine all layers of the model to find the final prediction layer that needs training. Use "trainable_variables" on the model to obtain all prediction layer variables, locate the layer where the predicted bounding box coordinates and class labels need fine-tuning, and set the variables individually.

[0101] Define a function for training one sample. This involves reading a tensor (an n-dimensional numerical array) image, performing data preprocessing in the model, and storing the preprocessed result and the actual tensor size. The result is then converted into a data format readable by the prediction tools in the object detection API and input into the prediction function to obtain predicted bounding boxes and their categories. A loss function is constructed to track the training process. By comparing the predicted bounding boxes with the ground truth image information, the loss function is established. The loss entropy is calculated using the loss tool in the object detection API, and the change rate is used to optimize the parameter weights in the model. The function for training one sample updates the model's parameter weights during the process and ultimately returns the loss value.

[0102] Step 1114: Set tracking parameters during training to select the optimal model as the first neural network model.

[0103] Combining the above equations and parameters, the model is trained according to the defined number of training iterations, and the best-performing model is obtained based on the metrics set during training (such as loss function, precision, or recall) and used as the first neural network model.

[0104] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the process for constructing the second neural network model provided by the present invention. In step 110 above, the step of constructing the second neural network model for training target tracking based on the deepSORT algorithm includes:

[0105] Step 112: Train the deepSORT tracker using the cosine_metric_learning training repository of the deepSORT algorithm, and export the training results using the TensorFlow framework. The exported training results are frozen pb models.

[0106] TensorFlow is an open-source software designed entirely in Python. It can train and run deep neural networks and has applications in many scenarios, such as image recognition, handwritten digit classification, recurrent neural networks, word embeddings, natural language processing, video detection, and more. TensorFlow can run on multiple CPUs or GPUs, and it can also run on mobile operating systems (such as Android and iOS). Its architecture is flexible, highly scalable, and supports various network models.

[0107] TensorFlow has two versions, TensorFlow 1 and TensorFlow 2, which can be selected according to actual needs. However, freezing a pb model can be done within the TensorFlow 1 framework. For example, after training tens of thousands of steps, sometimes 48,000 steps may perform better than 520,000 steps. In this case, you can choose to freeze the model at 48,000 steps and generate the weight file and other files required by the model.

[0108] For example, a detector for identifying different categories (e.g., staff and other personnel) is trained using the deepSORT training repository `cosine_mettric_learning`. For instance, photos of authorized staff in the area are compiled and labeled, and then a dataset containing some negative samples is placed in the same path. The model for the reID module in deepSORT is trained using the `cosine_metric_learning` repository, a process that applies the TensorFlow framework. The result of this step is a frozen pb model.

[0109] Step 113: Use the model from which the training results are derived as the second neural network model.

[0110] It should be noted that the first and second neural network models constructed above utilize the Tensorflow framework and the Object Recognition API (Application Programming Interface) in the implementation process. The process of modifying and retraining the RetinaNet-based network is described. The object recognition categories can be: masks, helmets, people, heads, and other parts. Before implementation, a dataset containing a predetermined number of images (e.g., 5000) can be formed using a portion of open-source datasets and a portion of self-annotated datasets. Then, a portion (e.g., 1000 images) of this dataset is randomly selected and assigned to the validation set.

[0111] In addition, set up the environment required for training based on the Tensorflow framework. Configure the drivers required for the local GPU and the libraries and modules involved in Tensorflow training, and select the appropriate versions of CUDA, cuDNN, and Python for Tensorflow. Install the necessary dependencies and corresponding versions of the tools required during training.

[0112] Defining the category index dictionary for model training requires providing the first neural network model with the corresponding numeric IDs for each category. The category index dictionary can have different formats depending on the model type; this is done by finding a dictionary sample specifically defined for that network. The number of target categories the trained model needs to process is also defined. The prepared dataset is then preprocessed. Images and information from the dataset are input into the first neural network model in a standard format for training. This is primarily achieved by converting category labels into one-bit valid encodings. Then, all training set images, target bounding boxes, and category labels are converted into tensors.

[0113] For example, in step 110 above, before constructing the first neural network model and the second network model, a region delineation module algorithm is also included to delineate the monitoring area into custom four-corner regions (ROIs).

[0114] The purpose of work area delineation is to more accurately monitor and identify events in important areas, and to add constraints to the final personnel movement prediction model, thereby more reasonably judging abnormal behavior events in the work area. The steps for constructing the work area delineation module algorithm are as follows:

[0115] First, a frame is captured from the camera and used as a preset frame. The width and height of the camera video image are then obtained from this preset frame. Second, the four corners of the working area are defined using the mouse. Due to the module's built-in logic, mouse clicks retrieve and store the coordinates of the current point, with the corner coordinates selected in a counter-clockwise direction. Finally, the coordinate information of the working area and the relevant preset information from the camera image are stored in a yml file for later retrieval.

[0116] If the camera's monitoring location changes or the work area is redefined during subsequent applications, this module can be used to reselect the work area.

[0117] In summary, based on the first and second neural network models that have been constructed and trained, we can now move on to the practical application stage.

[0118] For example, before performing step 120 (i.e., obtaining the video stream of the preset detection area), it is necessary to initialize and set the paths of the first neural network model and the second neural network model, as well as load the model and load the coordinate point information of the working area.

[0119] For example, the initialization process includes setting up the algorithm application parameters and loading the model. The coordinate information of the storage working area (yml file) is read and stored. The storage path of the obtained object detection model is set, and the first neural network model trained based on RetinaNet is loaded. The second neural network model trained above for object classification in DeepSORT is set and loaded. Relevant parameters in DeepSORT are then configured. The setting of these parameters depends on the application scenario and will be determined during subsequent deployment of the DeepSORT algorithm.

[0120] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the detection process using the first neural network model provided by the present invention. In step 120 above, the step of acquiring a video stream of a preset monitoring area and inputting the pre-processed video stream frames into the trained first neural network model to obtain the target detection result for each target in the frame includes:

[0121] Step 121: Obtain the video stream of the preset monitoring area by connecting to the camera device.

[0122] For example, by accessing the RTSP real-time video stream from an IP camera, one can obtain basic parameters of the video stream flowing from the camera, such as the number of frames per second, the width and height of each frame, and the number of pixels.

[0123] Step 122: Initialize the tracker, set relevant parameters and calculate the cosine distance metric, and set the maximum cosine distance between targets in two frames of images in the tracker module to control the calculation of features of adjacent targets.

[0124] For example, the nn_matching tool in deepSORT is used to construct a baseline for measuring the correlation of detection boxes in consecutive frames. The maximum cosine distance between targets is used as the overall matching metric to track the detection box information and movement in each frame. The tracker is then initialized using deepSORT's tracker tool and the newly established baseline for measuring the correlation of detection boxes.

[0125] Step 123: Preprocess the video stream frame by frame, change each frame to the size processed by the first neural network model, and adjust the parameters according to different scenarios.

[0126] For example, if affected by lighting conditions, parameters may be adjusted to improve the model's prediction output.

[0127] Step 124: Input the frame image into the first neural network model to obtain the target detection result of the image.

[0128] For example, each frame of image is input into the first neural network model trained with RetinaNet to obtain the object detection result for that frame. This object detection result includes the coordinates of the bounding box of the object (e.g., object S) in that frame, the bounding box, and the confidence score, and records the time information and coordinates of the object's appearance for later indexing.

[0129] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the prediction process using the second neural network model provided by the present invention. In step 130 above, the step of inputting the target detection result of at least one target into the trained second neural network model to perform position prediction and obtain the trajectory information corresponding to the target includes:

[0130] Step 131: Create the corresponding track detection box based on the target detection results.

[0131] For example, if the target detection result includes multiple targets, a corresponding track detection box is created for each target. The Kalman filter motion variables of the track detection boxes are initialized; the initial state of the tracks is uncertain.

[0132] Kalman filtering provides an efficient and computationally achievable method to estimate the state of a process and minimize the mean square error of the estimate. It assumes that all variables of the state are random and follow a Gaussian distribution, with each variable having its own mean and variance (each variable represents uncertainty).

[0133] Step 132: Use the second neural network model to perform secondary classification on the target detection results, and convert the target features, target box coordinates, target box category and confidence score extracted from the target detection results into the data format input to the deepSORT tracker.

[0134] Step 133: Predict the position of the created track detection box based on the trajectory detected in the previous frames in the deepSORT tracker.

[0135] For example, non-maximum suppression is used in all bounding box information to filter and remove bounding boxes with low confidence. The Kalman filter algorithm predicts the position of a specific track based on the tracker path (i.e., trajectory) detected in the previous few frames of the tracker. The position prediction of the Kalman filter is divided into two stages: (1) predicting the position of the track at the next time step, and (2) updating the predicted position based on target recognition.

[0136] For example, step 133 above includes:

[0137] Step 1331: Predict the position of the track detection box at time t based on the position of the created track detection box at time t-1.

[0138] Specifically, the tracker uses a Kalman filter to predict the motion trajectory of targets that have appeared in the previous few frames, and then inputs the converted target box information from step 132 into the tracker for prediction and judgment in the next frame.

[0139] The formulas for predicting the position of the track detection box at time t-1 (i.e., the current frame) and the formula for the covariance are as follows:

[0140] The formula for predicting the position of the track detection box is as follows:

[0141] x′=Fx;

[0142] x represents the mean value of the target location information corresponding to the track detection box at time t-1, which is determined by the center position coordinates (x, y) of the target bounding box. x ,x y The width-to-height ratio r and height h, along with the velocity change values ​​corresponding to each track detection box, constitute the x (for example, x is represented by an 8-dimensional vector as x = [x]).x ,x y ,r,h,v x ,v y ,v r ,v h (The initial values ​​of each speed value, as well as the initial values ​​during subsequent operation, are all 0), x ′ Let F represent the mean value of the target location information corresponding to the track detection box at time t, and let F represent the state transition matrix.

[0143] The covariance formula for the predicted position of the track detection box is as follows:

[0144] P′=FPF T +Q;

[0145] P represents the covariance matrix of the track detection box at time t-1, Q represents the noise matrix of the system (representing the reliability of the entire system), and P′ represents the covariance matrix of the track detection box at time t.

[0146] Step 1332: Based on the detected position at time t, update the positions of other track detection boxes associated with it to obtain the trajectory information corresponding to the target.

[0147] For example, step 1332 includes:

[0148] Step 13321: Calculate the error between the target detection result and the mean value of the track detection box at time t (i.e., the next frame). The calculation formula is as follows:

[0149] y = z - Hx′;

[0150] Where z represents the mean vector of the track detection box, excluding velocity change values, i.e., z = [x x ,x y [,r,h], where H represents the measurement matrix, which maps the mean vector x′ of the track to the detection space, and y represents the error value;

[0151] The mean vector x and covariance matrix P of the track detection box after state update are obtained using the following formulas:

[0152] S=HP′H Y +R;

[0153] x=x′+P′H T S -1 y;

[0154] P=(IP′H T S -1 H)P';

[0155] Where S represents the intermediate variable, I represents the identity matrix, and R represents the noise matrix of the deepSORT tracker. It is a 4x4 diagonal matrix, and the values ​​on the diagonal are the two coordinates of the center point and the noise of the width and height, respectively. It is initialized with arbitrary values, and the noise of the width and height is generally set to be greater than the noise of the center point. The formula first maps the covariance matrix P′ to the detection space, and then adds the noise matrix R.

[0156] Step 13322: Based on the updated track detection box, perform cascade matching between the current track detection box and its associated track detection boxes.

[0157] Specifically, the obtained predicted trajectory target information is matched with the actual target bounding box information in the current frame using the Hungarian algorithm. The update function in the tracker tool of DeepSORT is then used to update the information in the current tracker, resulting in successfully matched target bounding boxes and an updated tracker. For each track in the current tracker, firstly, tracks that do not match or have not been updated for a certain number of frames are filtered out. Then, the remaining valid tracks are converted into target bounding box data format, and the box is assigned a color and a sequence number using the track_id tool. This process is repeated until all boxes within a frame have undergone secondary identification, thus completing the cascaded matching.

[0158] Step 13323, and based on the result of the cascaded matching, output the coordinate frame position point of the target as the trajectory information of the target.

[0159] For example, the coordinate frame locations of different IDs are recorded in the backend database to record the routes of different external intruders within the monitoring range, and then the corresponding conclusions and target route coordinates are output.

[0160] In summary, compared with the prior art, the beneficial efficiencies of the present invention are as follows:

[0161] First, a retrained model based on the RetinaNet network is applied to a target recognition model with uneven training samples. By using a focal loss function during retraining, the weights of easily distinguishable samples are reduced, allowing the model to focus more on differentiating difficult-to-distinguish samples. This is particularly effective in models detecting strangers or objects, addressing the challenge of identifying difficult targets due to imbalanced positive and negative samples. Furthermore, this model has a decisive impact on the track detection bounding box for the target ID in subsequent tracking algorithms.

[0162] Secondly, the present invention connects the target detection results detected by the RetinaNet model and the input end of the deepSORT algorithm through the detection box data conversion module, and improves the running speed of the entire system through the format conversion module.

[0163] Third, the present invention uses the target trajectory information obtained from the first neural network model based on the RetinaNet network and the second neural network model based on the deepSORT algorithm to determine the application scenario requirements through data processing and storage, such as recording the target movement line, route and target event occurrence, etc., thereby strengthening the security monitoring of sensitive work areas and the control of the movement of intruders in real time.

[0164] The target identification and tracking device provided by the present invention is described below. The target identification and tracking device described below can be referred to in correspondence with the target identification and tracking method described above.

[0165] Please refer to Figure 6 , Figure 6 This is a schematic flowchart of the target recognition and tracking device provided by the present invention. A target recognition and tracking device 600 includes a model building module 610, a target detection module 620, and a target prediction module 630.

[0166] For example, the model building module 610 is used to build a first neural network model for training target recognition based on the RetinaNet network and a second neural network model for training target tracking based on the deepSORT algorithm.

[0167] For example, the target detection module 620 is used to acquire a video stream of a preset monitoring area and input the pre-processed video stream frames into a trained first neural network model to obtain the target detection result of each target in the frame.

[0168] For example, the target prediction module 630 is used to input the target detection result of at least one target into a trained second neural network model to perform position prediction and obtain the trajectory information corresponding to the target.

[0169] For example, the model building module 610 is further configured to:

[0170] A video processing module and a detection algorithm module are constructed using transfer learning based on the RetinaNet network to build the training environment for the first neural network model.

[0171] The training environment includes extracting image features using the ResNet and FPN networks in the RetinaNet network, searching for bounding boxes using anchor boxes, calling the Focal loss function in the class subnet to predict the class, and using the box subnet to predict the bounding box coordinates and size.

[0172] For example, the model building module 610 is further configured to:

[0173] Modify the parameters of the configuration file of the preset model to adapt to the training requirements of the RetinaNet network. The parameters include one or more combinations of the number of training target categories, the resized training images, the classification model, the training data path, the evaluation data path, and the label index path.

[0174] Retain the feature extraction layer and bounding box regression prediction layer of the RetinaNet network and freeze the object classification prediction layer that is not used in the RetinaNet network;

[0175] The training mode of the first neural network model is set and the training parameters are configured for model training. The training parameters include one or more combinations of the number of samples in one training session, the number of training batches, the learning rate, and the optimization method.

[0176] During training, tracking parameters are set to select the optimal model as the first neural network model.

[0177] For example, the model building module 610 is further configured to:

[0178] The deepSORT tracker is trained using the cosine_metric_learning training repository of the deepSORT algorithm, and the training results are exported using the TensorFlow framework. The exported training results are frozen pb models.

[0179] The model from which the training results are derived is used as the second neural network model.

[0180] For example, the target detection module 620 is further configured to:

[0181] By connecting to a camera device, a video stream of a preset monitoring area is obtained. The parameters of the video stream include the number of frames per second, the width and height of each frame, and one or more combinations of pixels.

[0182] Initialize the tracker, set relevant parameters and calculate the cosine distance metric, set the maximum cosine distance between targets in two frames of images using the tracker, and control the calculation of features of adjacent targets;

[0183] The video stream is preprocessed frame by frame, each frame of the image is changed to the size processed by the first neural network model, and the parameters are adjusted according to different scenarios.

[0184] The frame is input into the first neural network model to obtain the target detection result of the frame.

[0185] For example, the target prediction module 630 is further configured to:

[0186] Create a corresponding track detection box based on the target detection results;

[0187] The second neural network model is used to perform secondary classification on the target detection results, and the target features, target box coordinates, target box categories and confidence scores extracted from the target detection results are converted into the data format input to the deepSORT tracker.

[0188] The location of the created track detection box is predicted based on the trajectory detected in the previous frames of the deepSORT tracker.

[0189] For example, the target prediction module 630 is further configured to:

[0190] The position of the track detection box at time t is predicted based on the position of the created track detection box at time t-1.

[0191] Based on the detected position at time t, update the positions of other track detection boxes associated with it to obtain the trajectory information corresponding to the target.

[0192] For example, the target prediction module 630 is further configured to:

[0193] Determine the formulas for predicting the position of the track detection box and the covariance formula;

[0194] The formula for predicting the position of the track detection box is as follows:

[0195] x′=Fx;

[0196] x represents the mean value of the target location information corresponding to the track detection box at time t-1, which is determined by the center position coordinates (x, y) of the target bounding box. x ,x y It consists of aspect ratio r, height h, and velocity change values ​​corresponding to each track detection box, x′ represents the mean value of the target position information corresponding to the track detection box at time t, and F represents the state transition matrix;

[0197] The formula for the covariance of the predicted location of the detection box in this track is as follows:

[0198] P′=FPF T +Q;

[0199] P represents the covariance matrix of the track detection box at time t-1, Q represents the noise matrix of the system, and P′ represents the covariance matrix of the track detection box at time t.

[0200] For example, the target prediction module 630 is further configured to:

[0201] The error between the target detection result and the mean value of the track detection box is calculated using the following formula:

[0202] y = z - Hx′;

[0203] Where z represents the mean vector of the track detection box, H represents the measurement matrix, and y represents the error value;

[0204] The mean vector x and covariance matrix P of the track detection box are updated using the following formula:

[0205] S=HP′H Y +R;

[0206] x=x′+P ′ H T S -1 y;

[0207] P=(IP′H T S -1 H)P';

[0208] Where S represents the intermediate variable, R represents the noise matrix of the deepSORT tracker, and I represents the identity matrix;

[0209] Based on the updated track detection box, perform cascade matching between the current track detection box and its associated track detection boxes;

[0210] Based on the results of the cascaded matching, the coordinate frame position of the target is output as the trajectory information of the target.

[0211] It should be noted that the target recognition and tracking device provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0212] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute the target recognition and tracking method, the method including:

[0213] A first neural network model for training target recognition was constructed based on the RetinaNet network, and a second neural network model for training target tracking was constructed based on the deepSORT algorithm.

[0214] The video stream of the preset monitoring area is acquired, and the pre-processed video stream frames are input into the trained first neural network model to obtain the target detection result of each target in the frame.

[0215] The target detection results of at least one target are input into a trained second neural network model to predict the location and obtain the trajectory information corresponding to the target.

[0216] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the target recognition and tracking methods provided by the above methods.

[0218] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the target identification and tracking methods provided above.

[0219] The present invention provides an electronic device, a computer program product, and a processor-readable storage medium, wherein the computer program stored thereon enables the processor to implement all the method steps implemented in the above method embodiments and achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0220] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0221] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0222] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target recognition and tracking method, characterized in that, The method includes: Model construction steps: First, a first neural network model for training object recognition is constructed based on the RetinaNet network, including: retaining the feature extraction layer and bounding box regression prediction layer of the RetinaNet network, and freezing the unused object classification prediction layer in the RetinaNet network; defining a temporary object bounding box prediction checkpoint using `tf.train.checkpoint` to retain the tower layer and object bounding box prediction layer; then defining a temporary model checkpoint to retain the feature extraction layer and designating the object bounding box prediction layer as the temporary object bounding box prediction checkpoint; selectively recovering the weights of the feature extraction layer and bounding box regression prediction layer by specifying the path of the temporary model checkpoint from the checkpoints of the pre-trained model through a recovery function; Second, a second neural network model for training object tracking is constructed based on the deepSORT algorithm, including: training the deepSORT tracker using the `cosine_metric_learning` training repository and exporting the frozen pb model using the TensorFlow framework; Target detection steps: Acquire a video stream of a preset monitoring area, and input the pre-processed video stream frames into a trained first neural network model to obtain the target detection result for each target in the frame; input the target detection result of at least one target into a trained second neural network model to perform position prediction and obtain the trajectory information corresponding to the target.

2. The target recognition and tracking method according to claim 1, characterized in that, The steps for constructing the first neural network model for target recognition based on the RetinaNet network include: A video processing module and a detection algorithm module are constructed using transfer learning based on the RetinaNet network to build the training environment for the first neural network model; The training environment includes extracting image features using the ResNet and FPN networks in the RetinaNet network, searching for bounding boxes using anchor boxes, calling the Focal loss function in the class subnet to predict the class, and using the box subnet to predict the bounding box coordinates and size.

3. The target recognition and tracking method according to claim 2, characterized in that, The steps for transfer learning using the RetinaNet network include: Modify the parameters of the configuration file of the preset model to adapt to the training requirements of the RetinaNet network. The parameters include one or more combinations of the number of training target categories, the resized training images, the classification model, the training data path, the evaluation data path, and the label index path. The training mode of the first neural network model is set and the training parameters are configured for model training. The training parameters include one or more combinations of the number of samples in one training session, the number of training batches, the learning rate, and the optimization method. During training, tracking parameters are set to select the optimal model as the first neural network model.

4. The target recognition and tracking method according to claim 1, characterized in that, The steps for constructing a second neural network model for target tracking based on the deepSORT algorithm also include: The model from which the training results are derived is used as the second neural network model.

5. The target recognition and tracking method according to claim 1, characterized in that, The steps of acquiring a video stream of a preset monitoring area and inputting the preprocessed video stream frames into a trained first neural network model to obtain the target detection result for each target in the frame include: By connecting to a camera device, a video stream of a preset monitoring area is obtained. The parameters of the video stream include the number of frames per second, the width and height of each frame, and one or more combinations of pixels. Initialize the tracker, set relevant parameters and calculate the cosine distance metric, set the maximum cosine distance between targets in two frames of images using the tracker, and control the calculation of features of adjacent targets; The video stream is preprocessed frame by frame, each frame of the image is changed to the size processed by the first neural network model, and the parameters are adjusted according to different scenarios. The frame is input into the first neural network model to obtain the target detection result of the frame.

6. The target recognition and tracking method according to claim 1, characterized in that, The step of inputting the target detection result of at least one target into a trained second neural network model to perform position prediction and obtain the trajectory information corresponding to the target includes: Create a corresponding track detection box based on the target detection results; The second neural network model is used to perform secondary classification on the target detection results, and the target features, target box coordinates, target box categories and confidence scores extracted from the target detection results are converted into the data format input to the deepSORT tracker. The location of the created track detection box is predicted based on the trajectory detected in the previous frames of the deepSORT tracker.

7. The target recognition and tracking method according to claim 6, characterized in that, The step of predicting the position of the track detection box based on the trajectory detected in the previous frames of the deepSORT tracker includes: The position of the track detection box at time t is predicted based on the position of the created track detection box at time t-1. Based on the detected position at time t, update the positions of other track detection boxes associated with it to obtain the trajectory information corresponding to the target.

8. The target recognition and tracking method according to claim 7, characterized in that, The step of predicting the position of the track detection box at time t based on the position of the created track detection box at time t-1 includes: Determine the formulas for predicting the position of the track detection box and the covariance formula; The formula for predicting the position of the track detection box is as follows: ； This represents the mean value of the target location information corresponding to the track detection box at time t-1, which is determined by the center position coordinates of the target bounding box ( It consists of the aspect ratio r, height h, and the velocity change value corresponding to each track detection box. Let F represent the mean value of the target location information corresponding to the track detection box at time t, and let F represent the state transition matrix. The formula for the covariance of the predicted location of the detection box in this track is as follows: ； P represents the covariance matrix of the track detection box at time t-1, and Q represents the noise matrix of the system. Let represent the covariance matrix of the track detection box at time t.

9. The target recognition and tracking method according to claim 8, characterized in that, The step of correcting the positions of other track detection boxes associated with the detected position at time t to obtain the trajectory information corresponding to the target includes: The error between the target detection result and the mean value of the track detection box is calculated using the following formula: ； Where z represents the mean vector of the track detection box, H represents the measurement matrix, and y represents the error value; The mean vector x and covariance matrix P of the track detection box are updated using the following formula: ； ； ； Where S represents the intermediate variable, R represents the noise matrix of the deepSORT tracker, and I represents the identity matrix; Based on the updated track detection box, perform concatenated matching between the current track detection box and its associated track detection boxes; Based on the results of the cascaded matching, the coordinate frame position of the target is output as the trajectory information of the target.

10. A target recognition and tracking device, characterized in that, The device includes: The model building module is used to construct a first neural network model for training object recognition based on the RetinaNet network. This includes: retaining the feature extraction layer and bounding box regression prediction layer of the RetinaNet network, freezing the unused object classification prediction layer in the RetinaNet network; defining a temporary object bounding box prediction checkpoint using `tf.train.checkpoint` to retain the tower layer and object bounding box prediction layer; subsequently defining a temporary model checkpoint to retain the feature extraction layer and designating the object bounding box prediction layer as the temporary object bounding box prediction checkpoint; selectively recovering the weights of the feature extraction layer and bounding box regression prediction layer by specifying the path of the temporary model checkpoint from the checkpoints of the pre-trained model through a recovery function; and constructing a second neural network model for training object tracking based on the deepSORT algorithm. This includes: training the deepSORT tracker using the `cosine_metric_learning` training repository and exporting the frozen pb model using the TensorFlow framework. The target detection module is used to acquire the video stream of the preset monitoring area and input the pre-processed video stream frames into the trained first neural network model to obtain the target detection result of each target in the frame. The target prediction module is used to input the target detection results of at least one target into a trained second neural network model to predict the position and obtain the trajectory information corresponding to the target.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the target recognition and tracking method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the target recognition and tracking method as described in any one of claims 1 to 9.

Images