Multi-camera multi-target tracking method and device

Abstract

The application discloses a multi-camera multi-target tracking method and device, wherein cameras are separately arranged at different positions of a target area, each camera is provided with an independent database, and one common database is arranged, the method comprises the following steps: acquiring real-time videos of the cameras, and converting the real-time videos into image data of each frame according to time sequence; performing target prediction on the image data of each frame by using a pose estimation model, and extracting feature information of a target to be tracked according to a prediction result; and performing target matching and tracking on the feature information of the target to be tracked and information in the corresponding independent database or the common database by using an improved DeepSORT model, so as to obtain a tracking result. The application can improve the accuracy and robustness of pedestrian tracking in a multi-camera multi-target tracking system.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of target detection and target tracking, and particularly relates to a multi-camera multi-target tracking method and device. BACKGROUND

[0002] With the increasing adoption of multi-camera networks in various applications, the demand for multi-camera multi-object tracking (MCMOT) systems has surged, exceeding the use of single cameras. However, multi-camera tracking requires tracking targets across multiple camera views, which presents some unique challenges. Occlusion handling is one of the challenging tasks, but there has been relatively little theoretical development. This is mainly due to the complex relationships between objects and between objects and background, as well as the complexity of computational processing, as all possible partitions of the object set need to be considered theoretically. In a single view, useful prior information about the object of interest can be used to solve the occlusion problem. However, there are fundamental limitations to the functionality of single-view data. In contrast, the multi-view setup allows the use of additional information from multiple data to solve the occlusion problem, as an object that is occluded in one view may not be occluded in another view. Furthermore, from an information theory perspective, data from different views can reduce uncertainty about the object set of interest, thereby improving overall tracking performance.

[0003] Pose estimation models aim to infer the pose of a target by analyzing visual information in images or videos.

[0004] SORT (Simple Online and Realtime Tracking) is a simple, online, and real-time target tracking algorithm designed to achieve efficient target tracking with limited computational resources. The core idea of the SORT algorithm is to perform target tracking by combining target detection and motion prediction. The algorithm first uses a target detector (Faster R-CNN) to detect the position and bounding box of target objects in each frame. Then, a Kalman filter is used to model and predict the motion of the target. The Kalman filter combines the current state of the target and the motion model to estimate the position and velocity of the target in the next frame. SORT algorithm also introduces a data association method based on Hungarian algorithm, which is used to establish the association relationship between the current frame and the previous frame. This method matches the targets in the current frame with the tracked targets in the previous frame by minimizing the association cost, thereby determining the identity and trajectory of the target.

[0005] DeepSORT is a target tracking algorithm that is an extension of the SORT algorithm. DeepSORT combines the feature extraction capabilities of deep learning with traditional tracking techniques to improve the accuracy and robustness of target tracking. The algorithm first uses a convolutional neural network (CNN) to extract the feature representation of the target, and then uses these features for target detection and feature matching. DeepSORT introduces a representation method called "embedding" that can capture the appearance features of the target and combine them with the identity information of the target, thereby realizing long-term tracking of the target in a video sequence.

[0006] However, the traditional DeepSORT target tracking algorithm cannot be directly migrated in a multi-camera environment, and in a single-camera environment, a lot of information is lost due to occlusion, and the accuracy is not high when occlusion and cross motion occur. SUMMARY

[0007] To solve the above technical problems, the present application provides a multi-camera multi-target tracking method and device.

[0008] The first aspect of the present application provides a multi-camera multi-target tracking method, in which cameras are separately arranged at different positions of a target area, each camera is provided with an independent database, and a public database is provided. The method comprises:

[0009] Obtaining real-time videos of each camera and converting them into frame image data in time sequence;

[0010] Using a pose estimation model to predict targets for each frame of image data to obtain prediction results;

[0011] Using an improved DeepSORT model to track targets for each prediction result to obtain tracking results.

[0012] Further, the pose estimation model is used to predict targets for each frame of image data, and the specific process comprises:

[0013] Using a pose estimation model to predict targets for each frame of image data, and predicting the bounding box and joint position of each target in each frame of image data;

[0014] According to the bounding box of the target in each frame of image data, the appearance feature information of each target in each frame of image data is extracted by a ReID model;

[0015] According to the joint position of the target in each frame of image data, the position feature information of each target in each frame of image data is obtained by Zhang Zhengyou calibration method;

[0016] Based on the front and rear frame images taken by the same camera, the bounding box of each target in each frame image is predicted by a pose estimation model, and the bounding box is input into a Kalman filter to obtain the motion feature information of each target.

[0017] The prediction result is composed of the appearance feature information, the position feature information and the motion feature information of the targets.

[0018] Further, the improved DeepSORT model is used for target matching and tracking, and the specific process includes:

[0019] S301: According to the human joint position of each target in each frame image, the human joint confidence is calculated, and whether each target to be matched in each frame image is in a shielding state is judged according to the human joint confidence;

[0020] S302: If each target to be matched in each frame image is not shielded, a first cost matrix is calculated according to the appearance feature information and the position feature information of each target, and then Hungarian matching is performed to obtain a matching result;

[0021] S303: If each target to be matched in each frame image is shielded, a second cost matrix is calculated according to the appearance feature information and the motion feature information of each target, and then Hungarian matching is performed to obtain a matching result.

[0022] The second aspect of the present application proposes a multi-camera multi-target tracking device, which comprises:

[0023] An acquisition module is configured to acquire real-time videos of each camera and convert the real-time videos into frame image data in time sequence;

[0024] A target detection module is configured to perform target prediction on each frame image data to obtain a prediction result;

[0025] A feature extraction module is configured to extract feature information of a target from the prediction result;

[0026] A target tracking module is configured to perform target tracking on the feature information of each target and the feature information in a corresponding independent database or a public database to obtain a tracking result;

[0027] A public database is configured to store all targets and their feature information in the independent databases of the cameras that are considered to have been tracked;

[0028] An update module is configured to update the corresponding independent databases and the public database of each camera according to the tracking result;

[0029] An output module is configured to output the tracking result.

[0030] The beneficial effects of the present invention are as follows: the present invention separately sets cameras at different positions in the target area and uses a posture estimation model to improve the accuracy of target detection results; the improved DeepSORT model is used to perform multi-target tracking on each prediction result, which can effectively solve the problem of target occlusion and significantly improve the accuracy of target tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is an example diagram of an application scenario of the present invention;

[0032] Figure 2 Schematic diagram of the overall process of an embodiment of the present invention;

[0033] Figure 3 is a flowchart of steps in an embodiment of the present invention;

[0034] Figure 4 This is a target matching flow chart in an embodiment of the present invention;

[0035] Figure 5 Schematic diagram of the information interaction process between the independent database corresponding to each camera and the public database in an embodiment of the present invention;

[0036] Figure 6 Schematic diagram of the structure of a multi-camera multi-target tracking device in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] Figure 1 This is an example diagram of the application scenario of the present invention. Figure 1 As shown, multiple cameras are set up at different locations in the target area (such as classrooms, offices, office halls, etc.) Figure 1 In the figure, C1, C2, Ck, and Cn all represent cameras. During this period, different targets (pedestrians) will randomly appear and move around in the target area. Multiple cameras perform real-time video capture, target detection, and target tracking of the targets (pedestrians) from different directions. This can effectively solve the problem of inaccurate target tracking due to occlusion.

[0039] Figure 2 Schematic diagram of the overall process of an embodiment of the present invention. Figure 2In an embodiment, a plurality of cameras are arranged at different positions of a target area, and n cameras (C1-Cn) are arranged to capture real-time videos, and each camera is configured with a database (D1-Dn). The embodiment includes a target detection stage and a target tracking stage. In the target (pedestrian) detection stage, a picture with labels is obtained by a pose estimation pedestrian detector for each frame of image, and appearance feature information of a plurality of targets (pedestrians) is obtained by a ReID (re-identification) model, motion feature information of the plurality of targets (pedestrians) is obtained by a Kalman filter based on front and back frames of image captured by the same camera, and position feature information of the plurality of targets (pedestrians) is obtained by Zhang Zhengyou calibration method. In the target (pedestrian) tracking stage, the feature information (including the appearance feature information, the motion feature information and the position feature information) of the current frame of image is compared with the feature information of the last frame of image in each database and a public database, a best matching result is obtained, and a tracking result is output.

[0040] Figure 3 A step flowchart of the embodiment is shown.

[0041] The embodiment proposes a multi-camera multi-target tracking method, as shown in Figure 2 、 3 a camera is arranged at each of different positions of a target area, an independent database is arranged for each camera, and one public database is arranged, and the method includes the following steps.

[0042] S1: Real-time videos of each camera are obtained, and each frame of image data is converted in time sequence.

[0043] S2: A pose estimation model is used to predict a target for each frame of image data, and feature information of a target to be tracked is extracted according to a prediction result.

[0044] S3: An improved DeepSORT model is used to match and track the target to be tracked based on the feature information and information in a corresponding independent database or the public database, and a tracking result is obtained.

[0045] In an embodiment, before S1, a Zhang Zhengyou camera calibration method is used to calculate internal and external parameters of each camera, so as to prepare for subsequent conversion of camera coordinates into pixel coordinates.

[0046] Specifically, the Zhang Zhengyou camera calibration method specifically includes using a checkerboard calibration board to capture a plurality of images, and calculating the internal and external parameters of each camera through corner point detection and three-dimensional reconstruction.

[0047] An internal parameter matrix K of each camera is represented as:

[0048]

[0049] where f x and f y denote the focal length of the camera, (c x ,c y ) denote the coordinates of the principal point.

[0050] The extrinsic matrix of the camera [R|t] is expressed as:

[0051]

[0052] where r ij denote the elements of the rotation matrix R, t i denote the elements of the translation vector t. R is an orthogonal matrix satisfying R T R = I and det(R) = 1.

[0053] In S1, the embodiments of the present application can set up cameras (a total of n) at different positions in the target area according to different actual application scenarios and needs, and each camera is provided with a database. The n cameras are provided with 1 common database. The present application is aimed at the application scenario of multiple cameras simultaneously shooting the same target area. In order to improve the accuracy of target tracking and maintain the ID consistency between multiple cameras, 1 common database is set up. The common database is consistent with the single camera database in structure, but its target is to store the information of the tracked target from a global perspective. The common database can be considered as a collection of targets and their feature information that are considered to be correctly tracked in all camera individual databases.

[0054] In some preferred embodiments, in S2, a pose estimation model (i.e. a pose estimation pedestrian detector) is used to predict targets for each frame of image data, and the prediction obtains the bounding box, body joint position and confidence of each target in each frame of image. The bounding box, body joint position and confidence of each target in each frame of image constitute the result of target prediction.

[0055] The pose estimation model in the embodiments of the present application combines target detection and human key point detection (pose estimation), and is specially designed for detecting and identifying the key joint positions of human body in images. The pose estimation model can process the input image data in real time, quickly identify multiple joint points of human body such as head, shoulder, elbow, wrist, hip, knee and ankle, etc., and provide strong pose estimation capability for various applications; it has a wide range of applications in target motion capture, interactive systems, motion analysis and other fields, and is widely concerned by the industry due to its high efficiency and accuracy.

[0056] Specifically, the pose estimation model can use common pose estimation models such as YOLOv7-pose, OpenPose, AIphaPose, etc.

[0057] The specific process of the target prediction includes:

[0058] S101: Preprocess each input image data, adjust its size and normalize it. This can make the preprocessed image data more consistent with the input requirements of the pose estimation pedestrian detector, so as to better extract features.

[0059] S102: Extract features from the preprocessed image data using the deep convolutional neural network of the ReID model to obtain pedestrian appearance features.

[0060] S103: Perform target detection based on the extracted image features to predict the bounding box of the pedestrian in the image and determine the position of the pedestrian in the image.

[0061] Specifically, on the basis of feature extraction, the pose estimation model will simultaneously perform target detection and pose estimation tasks. By applying a series of convolution operations and prediction heads on the feature map, the model will predict the bounding box of the pedestrian target that may exist in the image and the position of the human joints of the pedestrian and their confidence, determining the approximate position of the pedestrian in the image.

[0062] Specifically, the pose estimation model will further predict the position of the joints of the pedestrian target in the image. This is achieved by predicting the coordinates of multiple key points on the feature map, which usually correspond to the joints of the human body, such as the head, shoulders, elbows, wrists, hips, knees, and ankles. Non-maximum suppression is performed on the joint position output by the model, and finally the joint position and its confidence of the pedestrian in the image are obtained. In some preferred embodiments, in S2, according to the bounding box and the human joint position of each target in each image, features are extracted, and the specific process includes:

[0063] S201: According to the bounding box of the target in each image, extract the appearance feature information of each target in each image through the ReID (re-identification) model.

[0064] Specifically, the appearance features extracted by the pedestrian re-identification (re-identification, ReID) model are visual features used to distinguish different individual pedestrians. These features are extracted from pedestrian images or video frames and can represent the uniqueness of pedestrians in appearance, such as color, texture, shape, etc.

[0065] S202: According to the human joint position of the target in each image, obtain the position feature information of each target in each image through Zhang Zhengyou calibration method.

[0066] Specifically, the position feature information refers to the physical position of the target in the region, that is, the position of the target relative to the coordinate origin after selecting a coordinate axis. Ideally, the position feature information calculated by each camera for the same target is the same, but in practice, it is affected by the position extraction error and calculation error, and there is an error in the position calculated by each camera for the same target, which can be controlled within 20 cm.

[0067] The Zhang Zhengyou calibration method is used to realize the conversion from the camera coordinate to the pixel coordinate:

[0068] By using the internal parameter matrix K and the external parameters R and t of the camera, the pixel point in the camera coordinate system can be converted into the pixel point in the pixel coordinate system.

[0069]

[0070] wherein [R|t] represents the external parameter matrix of the camera. Px and Py usually represent the position of the pixel point in the plane perpendicular to the optical axis of the camera, and Pz represents the distance of the pixel point to the lens of the camera. u and v are respectively the horizontal and vertical pixel coordinates of the pixel point on the image plane (i.e. the captured image), and w is usually 1 in the homogeneous coordinate representation, which is used for convenient matrix operation for coordinate transformation.

[0071] The standard Kalman filter is used for state estimation, and an 8-dimensional vector is used wherein (u, v) represents the center point of the object bounding box, r and h represent the length ratio and height of the bounding box respectively, and γ represents the change rate of them in the image, and the observation variable output by the Kalman filter is in the form of (u, v, γ, h).

[0072] The Kalman filter is used to predict the direction and speed of the position of the target detection box (i.e. the bounding box of the pedestrian), which is used to describe the motion state of the target detection box (i.e. the bounding box of the pedestrian) on the two-dimensional image. Therefore, the motion feature information can only be used on a single camera, because the angles of each camera are different, and the motion state of the detection box of the same target in different cameras is different.

[0073] S203: Based on the front and rear frame images captured by the same camera, the bounding box of each target in each frame image is obtained by predicting the pose estimation model, which is input into the Kalman filter to obtain the motion feature information of each target.

[0074] ​Multi-Object Tracking: Obtain real-time video, get image sequence (i.e. image frame in time sequence), locate multiple targets of interest (i.e. target detection stage), and maintain individual ID information and record its trajectory between consecutive frames (i.e. target tracking stage). In the multi-object tracking process, according to the number of cameras used, multi-object tracking can be divided into: single multi-object tracking and cross-camera multi-object tracking.

[0075] The application is suitable for cross-camera multi-object tracking of target area, and is based on the appearance feature information, position feature information and motion feature information of each target in each image obtained in the above target detection stage, and the feature information of the target to be tracked is formed.

[0076] Figure 4 The target matching flowchart in the embodiment of the application.

[0077] In some preferred embodiments, referring to Figure 4 As shown in the figure, the improved DeepSORT model is used for target matching and tracking, and the specific process includes:

[0078] S301: According to the position of the human joint of each target in each image, the human joint confidence is calculated, and whether each target to be matched in each image is in the occlusion state is judged according to the human joint confidence.

[0079] For example, the embodiment of the application uses the ankle of each target (pedestrian) in each image as the position of the human joint to make pose estimation of the target (pedestrian).

[0080] The parameters learned by the pose estimation model in the training process will output a set of predicted values for each detected joint node in the output layer after processing the input image, including the coordinate position of the joint node and the corresponding confidence score. The model learns and analyzes the image features through the convolutional neural network, and predicts the probability of the existence of each joint node at its corresponding position, i.e. the probability of the existence of the object. This probability reflects the confidence degree of the model that the joint node is indeed the real human joint node, and its value is between 0 and 1.

[0081] DeepSORT uses motion features and appearance features to match and track targets, but when targets overlap and cross, motion features become unreliable, so the application introduces position feature information into data association matching to calculate the position distance between targets.

[0082] Specifically, based on the appearance feature information, motion feature information and position feature information of each target obtained in the target detection stage, target matching and tracking are performed between each image in the target tracking stage.

[0083] In the target tracking stage, in terms of motion feature information, Mahalanobis distance is used to calculate the correlation between the to-be-tracked target and the tracked target of the Kalman filter, and the calculation formula is as follows:

[0084]

[0085] wherein d (1) (i,j) is the motion feature information measurement between the boundary box of the jth to-be-detected target and the ith detected target, d j is the predicted boundary box of the jth to-be-detected target, y i is the boundary box of the ith detected target, S i -1 is the covariance matrix of the two, is the indicator of this distance measurement, that is, when d (1) (i,j)≤t (1) is assigned a value of 1, otherwise a value of 0.

[0086] In the target tracking stage, in terms of appearance feature, cosine distance is used to measure the similarity between two appearance features, and the calculation formula is as follows:

[0087]

[0088] wherein d (2) (i,j) is the minimum cosine distance between the appearance feature vector of the jth to-be-detected target and the appearance feature vector of the ith detected target, r j is the appearance feature vector of the jth to-be-detected target, is the appearance feature vector of the ith detected target, is the indicator of this distance measurement, that is, when d (2) (i,j)≤t (2) is assigned a value of 1, otherwise a value of 0.

[0089] In the target tracking stage, in terms of position feature information, the distance between two points is used to calculate the physical distance between the tracked target and the to-be-tracked target, and the calculation formula is as follows:

[0090]

[0091] wherein d (3) (i,j) represents the minimum distance between the position of the jth to-be-detected target and the position of the ith detected target, is the x-coordinate of the ith to-be-tracked target, is the y-coordinate of the ith to-be-tracked target, is the x-coordinate of the ith tracked target, is the y-coordinate of the ith tracked target.x and L y is a database of x and y coordinates of tracked targets, is an indicator of this distance measure, i.e. is assigned the value 1 when d (3) (i,j)≤t (3) and 0 otherwise.

[0092] S302: If each target to be matched in each frame is not occluded, a first cost matrix is calculated according to the appearance feature information and the position feature information of each target, and then Hungarian matching is performed to obtain a matching result.

[0093] Preferably, the S302 is specifically:

[0094] S302a: If each target to be matched in each frame is not occluded, the appearance feature information and the position feature information of the target to be matched are matched with the feature information in the public database by Hungarian matching.

[0095] S302b: If the matching with the feature information in the public database is successful, the ID of the target is matched, and the ID, the appearance feature information and the position feature information of the target are updated to the public database and the corresponding independent database.

[0096] S302c: If the matching with the feature information in the public database is unsuccessful, the appearance feature information and the position feature information of the target to be matched are matched with the feature information in the corresponding independent database, if the matching is successful, the ID of the target to be matched is matched, and the ID, the appearance feature information and the position feature information of the target are updated to the corresponding independent database; if the matching is unsuccessful and a condition is met, the information of the target is removed.

[0097] The removal condition can be that the same target (pedestrian) is not successfully matched for k frames (k≥40) in succession.

[0098] Preferably, in S302, a final measurement value of position feature information and appearance feature information is calculated according to the appearance feature information and the position feature information between each target to be tracked and each tracked target, and a first cost matrix is constructed according to the final measurement value of position feature information and appearance feature information, and the final measurement value of position feature information and appearance feature information is specifically represented as:

[0099]

[0100] wherein, represents the final measurement value of position feature information and appearance feature information, d (2) (i,j) represents the minimum cosine distance between the appearance feature vector of the jth target to be matched and the appearance feature vector of the ith matched target, d (3)(i,j) represents the minimum distance between the position of the jth target to be matched and the position of the ith matched target, λ (2) represents the second custom weight coefficient.

[0101] This step uses the first cost matrix which combines position feature information and appearance feature information to perform Hungarian matching, reduces the problem of long-distance identity exchange caused by the appearance of each target (pedestrian) being too similar (which is manifested as the appearance feature information of different targets being greater than a threshold), and is conducive to improving the accuracy of target tracking.

[0102] S303: If each target to be matched in each frame of image is occluded, a second cost matrix is calculated according to the appearance feature and motion feature information of each target, and then Hungarian matching is performed to obtain a matching result.

[0103] Preferably, the S303 specifically includes:

[0104] S303a: If each target to be matched in each frame of image is occluded, the appearance feature information and motion feature information of the target to be matched are matched with the feature information in the corresponding independent database.

[0105] S303b: If the matching with the feature information in the corresponding independent database is successful, the ID of the target is matched, and the ID, appearance feature information and motion feature information of the target are updated to the corresponding independent database.

[0106] S303c: If the matching is unsuccessful and a removal condition is met, the information of the target to be matched is removed.

[0107] The removal condition can be that the same target (pedestrian) is not successfully matched for k frames (k≥40) in succession.

[0108] Preferably, in the S303, a final measurement value of motion feature information and appearance feature information is calculated according to the motion feature information and appearance feature information between each target to be tracked and each tracked target, and a second cost matrix is constructed according to the final measurement value of the motion feature information and appearance feature information. The final measurement value of the motion feature information and appearance feature information is specifically represented as:

[0109]

[0110] wherein, represents the final measurement value of the motion feature information and appearance feature information, d (1) (i,j) represents the motion feature information measurement between the jth target to be matched and the ith matched target, d (2) (i,j) represents the minimum cosine distance between the appearance feature vector of the jth target to be matched and the appearance feature vector of the ith matched target, λ (1)represents a first custom weight coefficient.

[0111] This step uses the second cost matrix which combines motion feature information and appearance feature information to perform the Hungarian matching, reduces the identity exchange problem caused by inaccurate position feature information extraction when the target is blocked, and is beneficial to improve the accuracy of target tracking.

[0112] In the original DeepSORT tracking algorithm, the algorithm is an algorithm only for a single camera, when a frame of image arrives, the information extracted in the frame is compared with the information in the last frame, and then tracking is realized. The independent database corresponding to the single camera in the embodiment of the application is a database for storing the information of the last frame. The scene to which the application is applied is that multiple cameras shoot the same area (i.e. target area) at the same time. In order to improve the accuracy of target tracking and maintain the ID consistency between multiple cameras, the application proposes the idea of public database.

[0113] The independent database corresponding to each camera stores the targets that have been successfully tracked in the video shot by the camera and related information, including appearance feature information, position feature information and motion feature information. Each camera additionally maintains its own single camera database for storing target information captured by the local camera and supplementing the public database.

[0114] The public database serves as the core of the whole system, which stores the target information (specifically, the appearance feature information and position feature information of the tracked targets) captured by all cameras and provides it to each camera for matching and updating. The public database is the union of the single camera databases and is only updated when the data source is reliable. The public database is maintained by all cameras.

[0115] The public database is consistent with the single camera database in structure, but its target is to store the information of the tracked targets from a global perspective, while each single camera database may only store part of the information of the tracked targets due to the blocking of the position, so when comparing with the information in the last frame, not only the information in the independent database of the corresponding camera is compared, but also the information in the public database is compared, so as to check and fill the missing information and improve the storage of all tracking target information in the whole system.

[0116] The first frame image is detected and the feature information of the target (pedestrian) is extracted, and the feature information of the target (pedestrian) is stored in the independent database of each camera, and the ID of the target (pedestrian) is randomly configured. Starting from the second frame image, the detected target (pedestrian) is extracted, and the extracted information is compared and matched with the information in the corresponding independent database and the public database, and the target (pedestrian) that is successfully matched is updated, that is, the information detected in this frame is used to replace the information in the single database and the public database; the target (pedestrian) that is not successfully matched is regarded as a new target (pedestrian) entering the field of view, and when a target (pedestrian) is not successfully matched for 40 frames, the corresponding information of the target (pedestrian) in the database is removed, and when the target (pedestrian) enters the field of view of the camera again, it will be regarded as a new target (pedestrian) and be randomly assigned an ID.

[0117] Figure 5 The information interaction process between the corresponding independent database and the public database of each camera in the embodiment of the application is shown in the schematic diagram.

[0118] Referring to Figure 5 The information interaction process between the corresponding independent database and the public database of each camera is shown in the schematic diagram.

[0119] The detected target (pedestrian) is extracted first, and the extracted information is compared and matched with the information in the independent database.

[0120] If the information of the target to be tracked in the corresponding independent database is matched successfully, the information in the corresponding independent database is updated and covered, and the information of the target in the public database is found according to the ID of the target in the corresponding independent single database, and the information of the target in the public database is updated and covered, so that the information of the target in the corresponding independent database and the public database is consistent.

[0121] If the target to be tracked in the corresponding independent database fails to match, the matching in the public database is attempted, and if the matching is successful, the information in the public database is updated and covered, and the information of the target in the single database is found according to the ID of the target in the public database, and the information of the target in the single database is updated and covered, so that the information of the target in the single database and the public database is consistent.

[0122] In the information sharing process between the independent database of each camera and the public database, when a certain camera cannot match the target in the public database, it will try to match through the information in the single camera database, thereby realizing information sharing and updating. This multi-camera information sharing method has high practicability and flexibility. By integrating the data of each camera, the target can be tracked more comprehensively. Due to the existence of the public database, even if a certain camera fails or data is lost, it can be supplemented and repaired through the data of other cameras, ensuring the robustness and reliability of the system.

[0123] Based on the same inventive concept, the embodiment of the present application proposes a multi-camera multi-target tracking device, which has the same or similar technical features as the embodiment of the multi-camera multi-target tracking method. The same or similar technical features will not be described again.

[0124] The embodiment of the present application proposes a multi-camera multi-target tracking device, as shown in Figure 6 The device comprises:

[0125] An acquisition module is configured to acquire real-time videos of each camera and convert them into image data of each frame in time sequence.

[0126] A target detection module is configured to perform target prediction on the image data of each frame to obtain a prediction result.

[0127] A feature extraction module is configured to extract feature information of the target from the prediction result.

[0128] A target tracking module is configured to perform target tracking on the feature information of each target with the feature information in the corresponding independent database or the public database, respectively, to obtain a tracking result.

[0129] A public database is configured to store the tracked targets and their feature information in all independent databases of the cameras.

[0130] An update module is configured to update the corresponding independent database and the public database of each camera according to the tracking result.

[0131] An output module is configured to output the tracking result.

[0132] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium, including ROM, RAM, magnetic disk or optical disk, etc.

[0133] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A multi-camera multi-target tracking method, characterized in that, The method comprises the following steps: Real-time videos of the cameras are acquired and converted into image data according to time sequence; A pose estimation model is used to predict targets in each frame of image data, and the boundary box, human joint position and confidence of each target in each frame of image data are predicted; appearance feature information of each target in each frame of image data is extracted through a ReID model according to the boundary box of the target in each frame of image data; position feature information of each target in each frame of image data is obtained through Zhang Zhengyou calibration method according to the human joint position of the target in each frame of image data; the boundary box of each target in each frame of image data is predicted through the pose estimation model based on the front and back frames of image data shot by the same camera, and the boundary box is input into a Kalman filter to obtain motion feature information of each target; the feature information of each target is composed of the appearance feature information, position feature information and motion feature information of the target; An improved DeepSORT model is used to match and track the feature information of the target to be tracked with the information in the corresponding independent database or the public database to obtain a tracking result, and the specific process of matching and tracking comprises the following steps: S301: The human joint confidence is calculated according to the human joint position of each target in each frame of image data, and it is judged whether each target to be matched in each frame of image data is in a shielding state according to the human joint confidence; S302: If each target to be matched in each frame of image data is not shielded, a first cost matrix is calculated according to the appearance feature information and position feature information of each target, and then Hungarian matching is performed to obtain a matching result; The calculation process of the first cost matrix comprises the following steps: the final measurement value of the position feature information and the appearance feature information is calculated according to the appearance feature information and the position feature information between each target to be tracked and each tracked target, and the first cost matrix is constructed according to the final measurement value; The final measurement value of the position feature information and the appearance feature information is specifically represented as: wherein, represents the final measurement value of the position feature information and the appearance feature information, d (2) (i,j) represents the minimum cosine distance between the appearance feature vector of the jth target to be matched and the appearance feature vector of the ith matched target, d (3) (i,j) represents the minimum distance between the position of the jth target to be matched and the position of the ith matched target, λ (2) represents the second custom weight coefficient; S303: If each target to be matched in each frame of image data is shielded, a second cost matrix is calculated according to the appearance feature information and the motion feature information of each target, and then Hungarian matching is performed to obtain a matching result.

2. The multi-camera multi-target tracking method of claim 1, wherein, In the target tracking process, in S302, if each target to be matched in each frame of image data is not shielded, the appearance feature information and the position feature information of the target to be matched are matched with the feature information in the public database through Hungarian matching; If the matching with the feature information in the public database is successful, the ID of the target is matched, and the ID, appearance feature information and position feature information of the target are updated into the public database and the corresponding independent database; If the matching with the feature information in the public database is unsuccessful, the appearance feature information and the position feature information of the target to be matched are matched with the feature information in the corresponding independent database, if the matching is successful, the ID of the target to be matched is matched, and the ID, appearance feature information and position feature information of the target are updated into the corresponding independent database; if the matching is unsuccessful and a condition is met, the information of the target is removed.

3. The multi-camera multi-target tracking method of claim 1, wherein, In the target tracking process, in S303, if each target to be matched in each frame of image is occluded, the appearance feature information and the motion feature information of the target to be matched are subjected to Hungarian matching with the feature information in the corresponding independent database; If the matching with the feature information in the corresponding independent database is successful, the ID of the target is matched, and the ID, the appearance feature information and the motion feature information of the target are updated to the corresponding independent database; If the matching is unsuccessful and the removal condition is met, the information of the target to be matched is removed.

4. The multi-camera multi-target tracking method of claim 1, wherein, In S303, the final measurement value of the motion feature information and the appearance feature information is calculated according to the motion feature information and the appearance feature information between each target to be tracked and the tracked target, and a second cost matrix is constructed according to the final measurement value, and the final measurement value of the motion feature information and the appearance feature information is specifically represented as: wherein, denotes the final measurement of motion feature information and appearance feature information, d (1) (i,j) denotes the motion feature information measurement between the bounding box of the jth target to be matched and the ith matched target, d (2) (i,j) denotes the minimum cosine distance between the appearance feature vector of the jth target to be matched and the ith matched target, λ (1) denotes the first custom weight coefficient.

5. A multi-camera multi-target tracking apparatus based on the multi-camera multi-target tracking method according to claim 1, characterized by, The device comprises: An acquisition module configured to acquire real-time videos of each camera and convert the real-time videos into frame image data in time sequence; A target detection module configured to perform target prediction on the frame image data to obtain a prediction result; A feature extraction module configured to extract feature information of a target from the prediction result; A target tracking module configured to perform target tracking on the feature information of each target with feature information in a corresponding independent database or a public database to obtain a tracking result; A public database configured to store all tracked targets and their feature information in the independent databases of all cameras; An update module configured to update the corresponding independent databases and the public database of each camera according to the tracking result; An output module configured to output the tracking result.

Images