Target tracking method and device

Abstract

The invention provides a target tracking method and device, and relates to the field of image processing. According to the embodiment of the invention, the method comprises the following steps: obtaining a first image and a second image which are synchronously collected by a first image collection device and a second image collection device which have partially overlapped visual angles, and respectively obtaining a plurality of first tracking detection frames and a plurality of second tracking detection frames for carrying out target tracking on the first image and the second image; mapping the plurality of second tracking detection frames into the first image to obtain corresponding mapping frames according to a mapping relation between the first image acquisition equipment and the second image acquisition equipment; and fusing the target objects in the overlapping region according to the intersection-to-union ratio between the plurality of first tracking detection frames and the plurality of mapping frames and the similarity degree of the appearance characteristics. Through the mapping relation between the two image acquisition devices, the same target object in the overlapping area of the first image and the second image is fused, and the accuracy of target tracking is improved.

Description

Target tracking method and device Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a target tracking method and device. Background Art

[0002] Multi-target tracking is to assign a tracking identifier to each target object in each frame of the video, so as to obtain the behavior trajectory of each target object based on the tracking identifier. Currently, the pedestrian re-identification algorithm (Re-identification, ReID) can be used to extract the appearance features of the tracked target object, and then the matching of the tracking targets in multiple image acquisition devices is completed by feature matching association. However, for two image acquisition devices with overlapping areas, due to the different acquisition perspectives of the two image acquisition devices, the same target will have different appearance features in the images captured by different image acquisition devices. For example, pedestrian 1 appears from the front in the acquisition perspective of image acquisition device A, but appears from the back in the acquisition perspective of image acquisition device B, resulting in a matching failure during feature matching association, causing the same target object to be assigned different tracking identifiers, resulting in inaccurate tracking results.

[0003] Summary of the Invention

[0004] The present invention provides a target tracking method and device to solve the deficiencies in the related art.

[0005] According to a first aspect of an embodiment of the present invention, a target tracking method is provided, the method comprising:

[0006] Acquire a first image and a second image synchronously captured by a first image capturing device and a second image capturing device, wherein the first image and the second image have an overlapping area;

[0007] Acquire a plurality of first tracking detection frames and a plurality of second tracking detection frames for performing target tracking on the first image and the second image respectively;

[0008] Mapping the plurality of second tracking detection frames to the first image to obtain corresponding mapping frames according to a mapping relationship between the first image acquisition device and the second image acquisition device;

[0009] Target objects in overlapping areas are fused according to intersection-over-union ratios (IoUs) and similarities in appearance features between the multiple first tracking detection frames and the multiple mapping frames.

[0010] In some embodiments, fusing the target objects in the overlapping area according to the intersection-over-union ratios and the similarity of appearance features between the multiple first tracking detection frames and the multiple mapping frames includes:

[0011] Determining a weight for matching each first tracking detection frame with each mapping frame according to an intersection-over-union ratio between each first tracking detection frame and each mapping frame, and a similarity in appearance features between each first tracking detection frame and each mapping frame;

[0012] Matching the plurality of first tracking detection frames with the plurality of mapping frames using a weighted bipartite graph matching algorithm;

[0013] If the first tracking detection frame matches the mapping frame, it is determined that the target object in the mapping frame and the target object in the first tracking detection frame are the same target object.

[0014] In some embodiments, the method further comprises:

[0015] acquiring a first tracking identifier assigned to each first tracking detection frame when performing target tracking on the first image, where the first tracking identifier is globally unique;

[0016] acquiring a second tracking identifier assigned to each second tracking detection frame when performing target tracking on the second image, where the second tracking identifier is globally unique;

[0017] When it is determined that the target object in the mapping frame and the target object in the first tracking detection frame are the same target object, the second tracking identifier of the second tracking detection frame corresponding to the mapping frame is replaced with the first tracking identifier of the first tracking detection frame.

[0018] In some embodiments, after replacing the second tracking identifier of the second tracking detection frame corresponding to the mapping frame with the first tracking identifier of the first tracking detection frame, the method further includes:

[0019] For other image pairs except the synchronized first frame image, obtaining a fusion result of the image pair, the image pair including the first image and the second image, the fusion result indicating whether the tracking identifier of the same target object in the first image and the second image is consistent;

[0020] If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and one of the tracking identifiers appears for the first time, then the tracking identifier that appears for the first time is replaced with the tracking identifier that has already appeared;

[0021] If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and each tracking identifier does not appear for the first time, the matching similarity between the tracking detection box and the prediction box of the target object in the previous frame image pair is obtained respectively, and the tracking identifier of the target object in the image with the large matching similarity is used to replace the tracking identifier of the target object in the other image.

[0022] In some embodiments, respectively acquiring a plurality of first tracking detection frames and a plurality of second tracking detection frames for performing target tracking on the first image and the second image includes:

[0023] Performing target detection on the first frame of the target image to obtain tracking detection frames of multiple target objects, creating a corresponding tracker and a feature library based on the tracking detection frame of each target object, and assigning a tracking identifier to each target object, the feature library being used to store appearance features of the target object, wherein the target image includes the first image and the second image;

[0024] For target images of other frames except the first frame, using each tracker to predict a prediction frame of the target object, and matching based on the intersection-over-union ratio between each prediction frame and each tracked detection frame, and the similarity between the first appearance feature of each target object in the feature library and the second appearance feature of each target object in the other frame target images;

[0025] For a tracking detection frame that matches the prediction frame, determining the tracking identifier of the prediction frame as the tracking identifier of the tracking detection frame in the target image of the other frame, updating a tracker using the position of the tracking detection frame, and updating the first appearance feature using the second appearance feature;

[0026] For a tracking detection frame that does not match the prediction frame, a corresponding tracker is created according to the tracking detection frame, a tracking identifier is assigned to the target object in the tracking detection frame, and the tracker that generates the prediction frame is deleted.

[0027] In some embodiments, the matching based on the intersection-over-union ratio between each predicted box and each tracked detection box, and the similarity between the first appearance feature of each target object in the feature library and the second appearance feature of each target object in the target image of the other frame includes:

[0028] Using a variety of target tracking algorithms to predict the prediction box of the target object;

[0029] Obtain the intersection-over-union ratio between the prediction boxes predicted by various target tracking algorithms and the tracking detection boxes;

[0030] Each predicted frame is matched with each tracking detection frame based on different intersection-over-union ratios and similarities between the first appearance feature in the feature library and the second appearance feature of the tracking detection frame.

[0031] In some embodiments, the method further comprises:

[0032] When performing target detection on the target image, if a specified target tracking algorithm among the multiple different target tracking algorithms predicts a prediction frame where a target object exists but fails to detect a tracking detection frame of the target object, then the prediction frame of the target object by the specified target tracking algorithm is determined as the tracking detection frame of the target object;

[0033] Obtaining intersection-over-union ratios between prediction boxes predicted using a plurality of different target tracking algorithms and each tracking detection box in the target image;

[0034] Based on different intersection-over-union ratios and similarities between the first appearance feature in the feature library and the second appearance feature of each tracking detection frame, matching is performed between the multiple prediction frames and the multiple tracking detection frames.

[0035] In some embodiments, the method further comprises:

[0036] Acquire a first calibration image and a second calibration image synchronously acquired by the first image acquisition device and the second image acquisition device;

[0037] detecting a first feature point of the size-invariant feature transformation in the first calibration image and a second feature point of the size-invariant feature transformation in the second calibration image;

[0038] Matching a first feature point in the first calibration image with a second feature point in the second calibration image to obtain a feature point pair;

[0039] A mapping relationship between the first image acquisition device and the second image acquisition device is determined according to the homogeneous coordinates of the first feature point in the feature point pair and the homogeneous coordinates of the second feature point in the feature point pair.

[0040] In some embodiments, after matching the first feature point in the first calibration image with the second feature point in the second calibration image to obtain a feature point pair, the method further includes:

[0041] In a specified area of ​​the first calibration image, obtaining a target first feature point constituting a feature point pair in the specified area;

[0042] In the second calibration image, obtaining target second feature points that match all target first feature points in the designated area;

[0043] Clustering all target second feature points in a set area, wherein the minimum value of the spacing between different set areas is greater than the size of the set area, and the size of the set area is equal to the size of the designated area;

[0044] Obtaining a target setting area with the largest number of target second characteristic points;

[0045] The feature point pairs where all second feature points of the targets in the target setting area are located are updated to the feature point pairs in the first calibration image and the second calibration image.

[0046] In some embodiments, acquiring the first image and the second image synchronously acquired by the first image acquisition device and the second image acquisition device includes:

[0047] Using a first stream pulling thread to obtain a first data stream collected by a first image acquisition device, and storing the first data stream in a first queue, using a second stream pulling thread to obtain a second data stream collected by a second image acquisition device, and storing the second data stream in a second queue;

[0048] Decoding the first data stream in the first queue using a first decoding thread to obtain a first image, and storing the first image in a third queue; decoding the second data stream in the second queue using a second decoding thread to obtain a second image, and storing the second image in the third queue;

[0049] If the synchronously captured second image or first image is not received for a set period of time after the first image or the second image is received, target tracking is performed on the received first image or the second image, and the third queue is cleared;

[0050] If the second image or the first image captured synchronously is received after a set time has passed since the first image or the second image was received, the first image and the second image are determined to be images captured synchronously by the first image acquisition device and the second image acquisition device, and the third queue is cleared.

[0051] According to a second aspect of an embodiment of the present invention, there is provided a target tracking device, the device comprising:

[0052] an acquisition unit, configured to acquire a first image and a second image synchronously acquired by a first image acquisition device and a second image acquisition device, wherein the first image and the second image have an overlapping area; and further configured to acquire a plurality of first tracking detection frames and a plurality of second tracking detection frames for performing target tracking on the first image and the second image, respectively;

[0053] a mapping unit, configured to map the plurality of second tracking detection frames to the first image to obtain corresponding mapping frames according to a mapping relationship between the first image acquisition device and the second image acquisition device;

[0054] The fusion unit is configured to fuse the target objects in the overlapping area according to the intersection-over-union ratios and the similarity of the appearance features between the multiple first tracking detection frames and the multiple mapping frames.

[0055] According to the above embodiments, the present invention obtains a first image and a second image synchronously acquired by a first image acquisition device and a second image acquisition device with partially overlapping perspectives, and respectively obtains multiple first tracking detection frames and multiple second tracking detection frames for target tracking of the first image and the second image; according to the mapping relationship between the first image acquisition device and the second image acquisition device, the multiple second tracking detection frames are mapped to the first image to obtain corresponding mapping frames; according to the intersection-and-union ratio between the multiple first tracking detection frames and the multiple mapping frames and the similarity of appearance features, the target objects in the overlapping area are fused. The multiple second tracking detection frames are mapped to the first image through the mapping relationship between the two image acquisition devices to obtain corresponding mapping frames, and based on the intersection-and-union ratio between the multiple first tracking detection frames and the multiple mapping frames and the similarity of appearance features, the same target object in the overlapping area of ​​the first image and the second image is fused, thereby improving the accuracy of target tracking.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0058] FIG1A is a schematic diagram showing a multi-target tracking result in a first frame of image according to an embodiment of the present invention.

[0059] FIG1B is a schematic diagram showing a multi-target tracking result in a tenth frame of image according to an embodiment of the present invention.

[0060] FIG2 is a schematic flow chart of a target tracking method according to an embodiment of the present invention.

[0061] FIG3A is a schematic flow chart of a method for implementing multi-target tracking in overlapping areas according to an embodiment of the present invention.

[0062] FIG3B is a schematic flow chart of a synchronous data acquisition method according to an embodiment of the present invention.

[0063] FIG4 is a schematic diagram showing a process of multi-target tracking according to an embodiment of the present invention.

[0064] FIG5A is a schematic diagram showing images captured by two image capture devices according to an embodiment of the present invention.

[0065] FIG5B is a schematic diagram showing feature point detection for an image according to an embodiment of the present invention.

[0066] FIG5C is a schematic diagram showing feature point matching for an image according to an embodiment of the present invention.

[0067] FIG6 is a schematic diagram showing how to remove matching error feature points by using spatial verification according to an embodiment of the present invention.

[0068] FIG. 7 is a schematic diagram showing coordinate conversion according to a mapping relationship according to an embodiment of the present invention.

[0069] FIG8 is a schematic diagram of a weighted bipartite graph matching algorithm according to an embodiment of the present invention.

[0070] FIG9A is a schematic diagram showing target tracking results in a first image acquisition device and a second image acquisition device according to an embodiment of the present invention.

[0071] FIG9B is a schematic diagram showing mapping target tracking results according to an embodiment of the present invention.

[0072] FIG9C is a schematic diagram showing a fused target tracking result according to an embodiment of the present invention.

[0073] FIG10 is a schematic diagram showing verification of a fusion result according to an embodiment of the present invention.

[0074] FIG11 is a schematic diagram of a tracking timing flow chart according to an embodiment of the present invention.

[0075] FIG12 is a schematic diagram of a target tracking device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0077] Figure 1A is a schematic diagram of the multi-target tracking results in the first frame image according to an embodiment of the present invention, and Figure 1B is a schematic diagram of the multi-target tracking results in the tenth frame image according to an embodiment of the present invention. As shown in Figures 1A and 1B, multi-target tracking is to assign a tracking identifier, also known as a tracking ID (Identity document), such as 4024, 4022, 4029, to the target object in each frame image in the video. Based on the tracking identifier of the target object, the behavior trajectory corresponding to each tracking identifier can be obtained.

[0078] Multi-target tracking algorithms can be applied to various areas of vision, such as security, autonomous driving, and healthcare. In security, they can be used to count the number of people in a specific area; in autonomous driving, they can be used to estimate the trajectories of pedestrians or vehicles; and in healthcare, they can be used to track the movement of cells. The target objects mentioned in this invention can be determined based on the application scenario; for example, the target objects can be vehicles, pedestrians, cells, etc.

[0079] In some scenarios, if the field of view of a single image acquisition device cannot cover an area of ​​interest, two or more image acquisition devices can be deployed to capture images of the area of ​​interest. When two or more image acquisition devices are used to capture images of an area of ​​interest, the captured images often have overlapping areas. The appearance features of the tracked target object can be extracted separately, and then used to achieve matching association between the target objects across multiple image acquisition devices.

[0080] However, the accuracy of using appearance features to match target objects is not high. The specific reasons include: for two image acquisition devices with overlapping areas, since the two image acquisition devices have different acquisition perspectives, the same target may have different appearance features in the images captured by different image acquisition devices. For example, user 1 appears in the front view in the acquisition perspective of image acquisition device A, but appears in the side view in the acquisition perspective of image acquisition device B. When using the appearance features of the front and the appearance features of the side for feature matching, it is easy to cause matching failure.

[0081] In view of this, the present invention provides a target tracking method, which fuses target objects in overlapping areas based on the coordinate mapping relationship between image acquisition devices to achieve multi-target tracking.

[0082] The following embodiments will illustrate a target tracking method provided by the present invention in conjunction with the accompanying drawings.

[0083] FIG2 is a flow chart of a target tracking method according to an embodiment of the present invention. As shown in FIG2 , the target tracking method includes the following steps 201 to 204 .

[0084] In step 201, a first image and a second image synchronously captured by a first image capturing device and a second image capturing device are acquired.

[0085] The first image and the second image have an overlapping area.

[0086] In this embodiment, the capture angle of view of the first image capture device partially overlaps with the capture angle of view of the second image capture device, that is, the first image captured by the first image capture device and the second image captured by the second image capture device have an overlapping area.

[0087] In one embodiment, a first image captured by a first image acquisition device and a second image captured by a second image acquisition device may be acquired respectively, and based on the acquisition time of the first image and the second image, the second image acquired synchronously with the first image may be acquired.

[0088] In step 202, a plurality of first tracking detection frames and a plurality of second tracking detection frames for performing target tracking on the first image and the second image are respectively obtained.

[0089] When the first image and the second image are acquired synchronously, target tracking can be performed on the first image to obtain multiple first tracking detection frames in the first image, and target tracking can be performed on the second image to obtain multiple second tracking detection frames in the second image.

[0090] In this embodiment, a multi-target tracking algorithm based on detection (tacking-by-detection) can be used to track targets in the first image and the second image respectively. For example, a SORT (Simple Online and Realtime Tracking) algorithm can be used to track targets in the first image and the second image respectively to obtain multiple first tracking detection frames and multiple second tracking detection frames.

[0091] In step 203, according to the mapping relationship between the first image acquisition device and the second image acquisition device, the plurality of second tracking detection frames are mapped into the first image to obtain corresponding mapping frames.

[0092] In this embodiment, the mapping relationship between the first image acquisition device and the second image acquisition device can be obtained in advance. When multiple first tracking detection frames of the first image and multiple second tracking detection frames of the second image are obtained, the multiple second tracking detection frames can be mapped to the image coordinate system where the first image is located according to the mapping relationship to obtain corresponding mapping frames.

[0093] In step 204, the target objects in the overlapping areas are fused according to the intersection-over-union ratios (IoUs) and the similarity of appearance features between the multiple first tracking detection frames and the multiple mapping frames.

[0094] In this embodiment, multiple first tracking detection frames can be matched with multiple mapping frames based on the intersection over union (IOU) and the similarity of appearance features. If the first tracking detection frame is successfully matched with the mapping frame, it means that the target object in the first tracking detection frame and the target object in the second tracking detection frame corresponding to the mapping frame are the same target object located in the overlapping area.

[0095] In this embodiment, if it is determined that a target object in a first image captured by a first image capture device is the same target object as a target object in a second image captured by a second image capture device, facial image information of the target object can be obtained and stored in association with a first tracking identifier of the target object in the first image capture device and a second tracking identifier of the target object in the second image capture device. In actual application, the associated facial image information, the first tracking identifier, and the second tracking identifier can be displayed simultaneously to prompt a user that the first tracking identifier and the second tracking identifier correspond to the same target object.

[0096] Those skilled in the art should understand that, in addition to the above display methods, the same identifier (eg, color) may be used to indicate that the first tracking identifier and the second tracking identifier represent the same target object, and the present invention is not limited thereto.

[0097] In summary, the present invention obtains a first image and a second image synchronously acquired by a first image acquisition device and a second image acquisition device with partially overlapping perspectives, and respectively obtains multiple first tracking detection frames and multiple second tracking detection frames for target tracking of the first image and the second image; according to the mapping relationship between the first image acquisition device and the second image acquisition device, the multiple second tracking detection frames are mapped to the first image to obtain corresponding mapping frames; according to the intersection-and-union ratio between the multiple first tracking detection frames and the multiple mapping frames and the similarity of appearance features, the target objects in the overlapping area are fused. The multiple second tracking detection frames are mapped to the first image through the mapping relationship between the two image acquisition devices to obtain corresponding mapping frames, and based on the intersection-and-union ratio between the multiple first tracking detection frames and the multiple mapping frames and the similarity of appearance features, the same target object in the overlapping area of ​​the first image and the second image is fused, thereby improving the accuracy of target tracking.

[0098] Before introducing each step, this embodiment will illustrate the overall concept of the present invention with reference to FIG3A .

[0099] FIG3A is a flow chart of a method for implementing multi-target tracking in an overlapping area according to an embodiment of the present invention. As shown in FIG3A , the method for implementing multi-target tracking in an overlapping area includes two parts:

[0100] 1. Acquire a mapping relationship between a first image acquisition device and a second image acquisition device whose acquisition viewing angles partially overlap. The mapping relationship in this embodiment refers to a homography matrix.

[0101] Acquire a first calibration image and a second calibration image synchronously acquired by a first image acquisition device and a second image acquisition device; detect a size-invariant feature transformation in the first calibration image

[0102] The invention relates to a method for determining a first feature point of a scale-invariant feature transform (SIFT) and a second feature point of a size-invariant feature transform in the second calibration image; matching the first feature point in the first calibration image and the second feature point in the second calibration image to obtain a feature point pair; and determining a mapping relationship between the first image acquisition device and the second image acquisition device according to the homogeneous coordinates of the first feature point in the feature point pair and the homogeneous coordinates of the second feature point in the feature point pair.

[0103] 2. Multi-target tracking, namely target detection and matching.

[0104] 1. Synchronous data collection;

[0105] The poses of the same target object presented by different image acquisition devices at the same time are most similar. To fuse the same target object in the overlapping area of ​​the two image acquisition devices, this embodiment obtains a first image and a second image captured synchronously by the first image acquisition device and the second image acquisition device. If the difference between the capture times of the first and second images is less than a set time interval, the first and second images are considered to be synchronously captured.

[0106] 2. Perform target detection and tracking on the collected synchronous data;

[0107] Target detection and positioning are performed on the synchronously acquired images using a target detection algorithm, such as Yolov5 or FasterRCNN. A tracker is created for each detection result using a Kalman filter model, and a tracking identifier is generated for each detection result in the corresponding image acquisition device.

[0108] 3. Tracking data fusion;

[0109] The tracking results in the second image acquisition device are mapped to the image coordinate system of the first image acquisition device through a mapping relationship, and matching associations are performed based on the intersection-and-union ratio between the tracking results in the first image acquisition device and the mapping results, as well as the appearance features, and the tracking identifiers of the associated target objects are unified, so that the tracking identifiers of the same target object are globally unique in the area collaboratively acquired by the first image acquisition device and the second image acquisition device.

[0110] 4. Verify the fused data.

[0111] In the first frame, because the target object's tracking identifier is initially created, the same target typically has different tracking identifiers. After a single fusion using the above steps, the same target object will have the same tracking identifier. The consistency of the tracking identifiers for the same target object across different image acquisition devices after fusion is used to verify the correctness of the fusion results in subsequent frames.

[0112] The following embodiments will introduce each step in detail.

[0113] In some embodiments, a multi-threaded approach can be used to acquire images synchronously captured by any two image acquisition devices with overlapping regions. That is, acquiring a first image and a second image synchronously captured by a first image acquisition device and a second image acquisition device can include steps 2011 to 2013.

[0114] In step 2011, a first stream pulling thread is used to obtain a first data stream captured by a first image acquisition device, and the first data stream is stored in a first queue. A second stream pulling thread is used to obtain a second data stream captured by a second image acquisition device, and the second data stream is stored in a second queue.

[0115] Figure 3B is a flow chart of a synchronous data acquisition method according to an embodiment of the present invention. As shown in Figure 3B, the first stream pulling thread 302 obtains the first data stream collected by the first image acquisition device 301, and stores the first data stream in the first queue 303. The second stream pulling thread 305 is used to obtain the second data stream collected by the second image acquisition device 304, and stores the second data stream in the second queue 306.

[0116] In step 2012, the first data stream in the first queue is decoded using a first decoding thread to obtain a first image, and the first image is stored in a third queue. The second data stream in the second queue is decoded using a second decoding thread to obtain a second image, and the second image is stored in the third queue.

[0117] As shown in Figure 3B, the first data stream in the first queue 303 is decoded using a first decoding thread 307 to obtain a first image 308, which is then stored in a third queue 309. The second data stream in the second queue 306 is decoded using a second decoding thread 310 to obtain a second image 311, which is then stored in a third queue 309. In one example, the first and second images can be stored in association with their acquisition times and the IDs of the corresponding image acquisition devices to facilitate querying information corresponding to the images.

[0118] In step 2013a, if the synchronously captured second image or first image is not received for a set period of time after the first image or the second image is received, target tracking is performed on the received first image or the second image, and the third queue is cleared.

[0119] As shown in FIG3B , in response to detecting that the first image or the second image has been received in the third queue, in which case the number of images in the third queue is 1 and does not match the number of image acquisition devices, a determination is made as to whether a set time has elapsed since the first image or the second image was received. If the set time has elapsed, a detection is made as to whether the synchronously acquired second image or the first image has been received. If the synchronously acquired second image or the first image has not been received within the set time, target tracking processing is performed on the received first image or the second image, and the third queue is cleared. The target tracking processing process will be described in detail in subsequent embodiments.

[0120] When the waiting time is longer than the set time, target tracking is performed on the received first image or second image. When network instability or abnormality occurs in one of the image acquisition devices, the other image acquisition device will not be affected, thereby ensuring the stability of the data flow.

[0121] In step 2013b, if the set time elapses after the first image or the second image is received and the second image or the first image is received synchronously, the first image and the second image are determined to be images synchronously captured by the first image acquisition device and the second image acquisition device, and the third queue is cleared.

[0122] If the second image or the first image is received after the set time, it is considered that the received first image and the second image are collected synchronously. Therefore, target tracking processing is performed on the synchronously collected first image and the second image respectively, and the third queue is cleared.

[0123] After the third queue receives the first image or the second image, all received images are output after a set time period, and the third queue is cleared to facilitate receiving the first image and the second image captured synchronously in the next frame.

[0124] When acquiring the synchronously captured first and second images using the above method, the time difference between the capture of the first image and the capture of the second image is less than a set duration. The set duration can be determined based on the frame rate of the image capture device. For example, the set duration is equal to 1000 / fps, where fps represents the frame rate of the image capture device. For example, if the frame rate of the first image capture device is 25, the time difference between the capture of the first image and the capture of the second image should be less than 40ms.

[0125] When a first image and a second image are acquired synchronously by a first image acquisition device and a second image acquisition device, target tracking can be performed on the first image to obtain multiple first tracking detection frames, and target tracking can be performed on the second image to obtain multiple second tracking detection frames. Considering that the target tracking process for the first image and the second image is the same, for ease of description, this embodiment uses the term "target image" to refer to the first image and the second image. The following embodiment will describe the target tracking process for the target image.

[0126] Target detection and appearance feature extraction of the target object are performed on the first frame target image. A tracker and a feature library are created for each target object based on the tracking detection frame obtained by target detection, and a tracking identifier is assigned to the tracking detection frame corresponding to the target object. The feature library is used to store the appearance features of the target object.

[0127] For target images of other frames except the first frame target image, target detection is performed on the target images of other frames to obtain multiple tracking detection frames, and the second appearance features of each target object are extracted. The created trackers are used to predict the prediction frames of each target object in the target images of other frames, and matching is performed based on the intersection-over-union ratio between each prediction frame and each tracking detection frame, as well as the similarity between the first appearance features of each target object in the feature library and the second appearance features of each target object.

[0128] For a tracking detection frame that matches a prediction frame, the tracking identifier of the prediction frame is determined as the tracking identifier of the tracking detection frame in the target image of another frame, and the tracker is updated using the position of the tracking detection frame, and the first appearance feature is updated using the second appearance feature.

[0129] For a tracking detection frame that does not match the prediction frame, a corresponding tracker is created according to the tracking detection frame, a tracking identifier is assigned to the target object in the tracking detection frame, and the tracker that generates the prediction frame is deleted.

[0130] In this embodiment, the age of each tracker is set to 1, that is, the tracker is deleted if it fails to match once, thereby reducing the error caused by wrong prediction.

[0131] The appearance features in this embodiment may be features of a Histogram of Oriented Gradient (HOG), FHOG, a color histogram, or a deep CNN. The feature library corresponding to the target object may store the appearance features of the target object in the first frame of the target image, as well as the appearance features of the target object in subsequent frames when the prediction frame and the tracking detection frame in the subsequent frames match. The first appearance feature may be determined based on multiple appearance features in the feature library, for example, it may be the average feature of multiple appearance features in the feature library, or the mean square error of multiple appearance features. The method of determining the first appearance feature in this embodiment may smooth out fluctuations in the appearance features, such as fluctuations in the appearance features caused by postures such as bending or squatting of the target object, and fluctuations in the appearance features caused by changes in the color of the target object due to light influences.

[0132] FIG4 is a flow chart of a multi-target tracking process according to an embodiment of the present invention. As shown in FIG4 , a tracker constructed based on a Kalman filter model can be used for prediction, and the prediction results are associated with the detection results, as well as the appearance features and the average features in the tracker, and the cost matrix is ​​calculated according to the following formula (1).

[0133] In formula (1), IOU represents the intersection-over-union ratio between the tracking detection box and the prediction box. mn represents the intersection-over-union ratio of the mth tracking detection box and the nth prediction box, m and n are positive integers, F represents the similarity of the appearance features between the tracking detection box and the prediction box, F mn represents the similarity between the appearance features in the mth tracking detection frame and the appearance features in the nth prediction frame, and w represents a weight parameter. In this embodiment, the appearance features in the prediction frame can be determined based on the first appearance features in the feature library.

[0134] This embodiment performs weighted fusion based on the motion information (i.e., the tracking detection frame) and appearance feature information of the target object, providing a basis for accurate matching of the target object in a single image acquisition device.

[0135] Based on the above cost matrix, a weighted bipartite graph matching algorithm is used to associate the tracking detection frame with the prediction frame predicted by the tracker. A tracker is created for the tracking detection frame that fails to match, and the unmatched tracker is deleted. For the matched tracker, the tracker is updated using the matched tracking detection frame, that is, the Kalman filter model is updated. The update can improve the tracker's prediction accuracy of the prediction frame's position in the next frame image, and the first appearance feature is updated using the feature averaging method. For example, the first appearance feature can be updated using formula (2).

[0136] In formula (2), V t represents the average feature, i.e. the first appearance feature, v i represents the i-th appearance feature that has been matched, and T represents the number of appearance features that have been matched. In one example, T can be set to 50.

[0137] In some embodiments, in order to address the prediction limitations of the Kalman filter model, for example, when the target object performs nonlinear motion or the target object in the current frame is not effectively identified and located (missed identification occurs), a single target tracking algorithm can be used for compensatory tracking, that is, a plurality of different target tracking algorithms are used to predict the prediction frame of the target object; the intersection-over-union ratios between the prediction frames predicted by the plurality of different target tracking algorithms and the tracking detection frames are obtained; and each prediction frame is matched with each tracking detection frame based on different intersection-over-union ratios and the similarity between the first appearance feature in the feature library and the second appearance feature of the tracking detection frame.

[0138] In this embodiment, in addition to using the Kalman filter model for prediction, the median optical flow tracking algorithm and the single target tracking algorithm based on correlation filtering can also be used for prediction. That is, the IOU correlation matrix of the other two prediction methods is added to the formula (1). Among them, the single target tracking algorithm based on correlation filtering can include KCF (Kernelized Correlation Filters), CSK, DCF, SRDCF, etc.

[0139] In one embodiment, a Kalman filter model, a median optical flow tracking algorithm, and a correlation filter KCF algorithm are used for prediction. In this case, the cost matrix can be calculated according to formula (3).

[0140] In formula (3), k represents the weight of the IOU cost matrix between the prediction of the Kalman filter model and the current frame detection, c represents the weight of the IOU cost matrix between the prediction of the correlation filter FCK and the current frame detection, and m is the weight of the IOU cost matrix between the median optical flow tracking algorithm and the current frame detection.

[0141] In practical applications, the above weight parameters can be adjusted according to the specific tracking scenario to ensure that the previous frame's prediction frame for the current frame and the tracking detection frame of the current frame are fully correlated and matched.

[0142] Those skilled in the art should understand that in addition to using the Kalman filter model, the median optical flow tracking algorithm and the correlation filter KCF algorithm for prediction, prediction can also be performed through a combination of other target tracking algorithms, and this embodiment does not limit this.

[0143] For a single target tracking algorithm, the position of the target object in the current frame is essentially predicted by the position of the target object in the previous frame. Therefore, when the target object in the current frame is not detected due to occlusion or deformation, the median optical flow tracking algorithm or the KCF algorithm considers more factors during prediction, and the prediction result is more accurate. Therefore, the prediction box predicted by the median optical flow tracking algorithm or the prediction box predicted by the KCF algorithm can be determined as the tracking detection box of the target object, that is, the prediction box is used instead of the tracking detection box. The Kalman filter model is updated with the replaced tracking detection box to prevent the Kalman filter model from affecting the prediction of the subsequent target object's location due to missed detection of the target object, thereby avoiding the jump of the target tracking identifier caused by missed detection of the target detector.

[0144] In some embodiments, when performing target detection on the target image, if a specified target tracking algorithm among multiple different target tracking algorithms predicts a prediction frame where a target object exists but fails to detect a tracking detection frame of the target object, the prediction frame of the target object by the specified target tracking algorithm is determined as the tracking detection frame of the target object;

[0145] Obtaining intersection-over-union ratios between prediction boxes predicted using a plurality of different target tracking algorithms and each tracking detection box in the target image;

[0146] Based on different intersection-over-union ratios and similarities between the first appearance feature in the feature library and the second appearance feature of each tracking detection frame, matching is performed between the multiple prediction frames and the multiple tracking detection frames.

[0147] The following embodiments will illustrate the calculation of the homography matrix of image acquisition devices with overlapping areas with reference to the accompanying drawings.

[0148] In this embodiment, the first calibration image and the second calibration image synchronously captured by the first image acquisition device and the second image acquisition device can be obtained through multi-threading, as shown in Figure 5A, which are images captured by the first image acquisition device and the second image acquisition device based on their respective perspectives.

[0149] SIFT feature points are detected in the first calibration image and the second calibration image respectively, as shown in FIG5B , and SIFT feature points are detected for each calibration image.

[0150] In one embodiment, the region of interest ROI in the first calibration image and the region of interest ROI in the second calibration image may be determined according to the overlapping area of ​​the two image acquisition devices, and then SIFT feature points in the ROI regions are respectively obtained.

[0151] When the first feature point in the first calibration image and the second feature point in the second calibration image are obtained, the feature points in the two calibration images can be matched. In this embodiment, a cross matching algorithm and / or a nearest neighbor matching algorithm can be used for feature point matching.

[0152] A cross-matching algorithm is used to match feature points, including: obtaining a second feature point in the second calibration image that is closest to the first feature point P1 in the first calibration image and whose distance is smaller than a threshold; conversely, obtaining a first feature point P1 in the first calibration image that is closest to the second feature point P2 in the second calibration image and whose distance is smaller than a threshold. If the two obtained results are the same, it means that the first feature point P1 matches the second feature point P2.

[0153] For example, the left image of Figure 5C is the first calibration image, and the right image of Figure 5C is the second calibration image, where the distance between a first feature point P1 in the first calibration image and a second feature point P2 in the second calibration image is the closest and is less than the threshold R, which means that the first feature point P1 and the second feature point P2 can be matched. Then, it is verified whether the second feature point P2 in the second calibration image has the closest distance to the first feature point P1 in the first calibration image and is less than the threshold R. If so, the first feature point P1 and the second feature point P2 are determined as a set of feature point pairs, that is, the first feature point P1 and the second feature point P2 are a pair of matching points.

[0154] The nearest neighbor matching algorithm performs feature point matching based on the principle that the similarity between the first feature point P1 and the second feature point P2 is greater than the similarity between the first feature point P1 and other feature points in the second calibration image.

[0155] The feature point matching is performed using a nearest neighbor matching algorithm, including: for a first feature point P1 in the first calibration image, obtaining two second feature points P2 and P3 closest to the first feature point P1 in the second calibration image, wherein the distance between the first feature point P1 and the second feature point P2 is D1, and the distance between the first feature point P1 and the second feature point P3 is D2. When D1 / D2 is less than a certain threshold, the first feature point P1 and the second feature point P2 are determined to be a set of feature point pairs.

[0156] In this embodiment, when the nearest neighbor matching algorithm is used for feature point matching, a smaller threshold can be set to retain more matching points. However, a smaller threshold may retain some incorrect matching points. Therefore, when the nearest neighbor matching algorithm is used for feature point matching, a spatial verification method can be used to remove incorrect matching points.

[0157] In order to further ensure the correctness of the matching, this embodiment adds spatial verification on the basis of cross matching and nearest neighbor matching. That is, after matching the first feature points in the first calibration image and the second feature points in the second calibration image to obtain feature point pairs, the embodiment also includes: in a specified area of ​​the first calibration image, obtaining target first feature points that constitute feature point pairs in the specified area; in the second calibration image, obtaining target second feature points that match all target first feature points in the specified area; clustering all target second feature points in a set area, with the minimum value of the spacing between different set areas being greater than the size of the set area, and the size of the set area being equal to the size of the specified area; obtaining a target set area with the largest number of target second feature points; and updating the feature point pairs where all target second feature points in the target set area are located to feature point pairs in the first calibration image and the second calibration image.

[0158] Figure 6 is a schematic diagram of using spatial verification to remove erroneous matching feature points according to an embodiment of the present invention. As shown in Figure 6, a specified area 601 is selected in the first calibration image 61, and the set size of the specified area 601 can be 20*20. In the second calibration image 62, target second feature points that match all target first feature points in the specified area are obtained; based on the position distribution of all target second feature points in the second calibration image, hierarchical clustering is performed according to pixel distance, and the minimum value of the class distance is greater than the size of the set area, for example, 20 pixels; the area where a category with the largest proportion of cluster points is located is obtained as the target set area, the feature point pairs in the target set area are retained, and the feature point pairs that match the midpoints of other categories are deleted, that is, the feature point pairs in category 1 are retained, and the feature point pairs in category 2 and category 3 are deleted.

[0159] When a successfully matched feature point pair is obtained, a RANSAC (RANdom SAmple Consensus) algorithm may be used to calculate a homography matrix H based on the matched feature point pair, thereby obtaining a mapping relationship between the first image and the second image.

[0160] In this embodiment, a coordinate transformation from the first image acquisition device to the second image acquisition device may be established, as shown in formula (4).

[0161] In formula (4), (x1, y1, 1) is the homogeneous coordinate of the first feature point in the feature point pair, (x2, y2, 1) is the image homogeneous coordinate of the second feature point in the feature point pair, and the homography matrix H is shown in formula (5).

[0162] In one embodiment, based on the mapping relationship between the first image acquisition device and the second image acquisition device, multiple first tracking detection frames can be mapped to the second image to obtain corresponding mapping frames. Target objects in the overlapping area can be fused based on the intersection-over-union ratios and similarity in appearance features between the multiple second tracking detection frames and the multiple mapping frames. By mapping multiple first tracking detection frames in the first image to the second image, and then mapping multiple second tracking detection frames in the second image to the first image, the maximum number of fused target objects can be obtained, thereby improving fusion accuracy.

[0163] For example, after mapping multiple first tracking detection frames in the first image to the second image for fusion, there are 4 target objects that can be fused, and after mapping multiple second tracking detection frames in the second image to the first image for fusion, there are 5 target objects that can be fused. In this case, it is chosen to map the multiple second tracking detection frames in the second image to the first image for fusion, thereby avoiding inaccurate fusion results due to occlusion between target objects in the first image.

[0164] In the aforementioned embodiment, each image acquisition device performs independent target tracking, resulting in a unique tracking identifier for the target object within a single image acquisition device. To ensure that the same target object in an overlapping region has a globally unique tracking identifier, the tracking data from each image acquisition device can be fused based on the mapping relationship. Tracking data fusion in this embodiment can be understood as merging the tracking results in the overlapping region, resulting in a unique tracking identifier for the same target object across both image acquisition devices.

[0165] In some embodiments, according to the intersection-and-union ratios (IoUs) and similarities in appearance features between the multiple first tracking detection frames and the multiple mapping frames, the target objects in the overlapping area are fused, which may include: determining the weight of matching each first tracking detection frame with each mapping frame according to the IoUs between each first tracking detection frame and each mapping frame, and similarities in appearance features between each first tracking detection frame and each mapping frame; matching the multiple first tracking detection frames with the multiple mapping frames using a weighted bipartite graph matching algorithm; if the first tracking detection frame matches the mapping frame, determining that the target object in the mapping frame and the target object in the first tracking detection frame are the same target object.

[0166] In some embodiments, fusing the target objects in the overlapping area includes:

[0167] acquiring a first tracking identifier assigned to each first tracking detection frame when performing target tracking on the first image, where the first tracking identifier is globally unique;

[0168] acquiring a second tracking identifier assigned to each second tracking detection frame when performing target tracking on the second image, where the second tracking identifier is globally unique;

[0169] When it is determined that the target object in the mapping frame and the target object in the first tracking detection frame are the same target object, the second tracking identifier of the second tracking detection frame corresponding to the mapping frame is replaced with the first tracking identifier of the first tracking detection frame.

[0170] Figure 7 is a schematic diagram of coordinate transformation based on a mapping relationship according to an embodiment of the present invention. As shown in Figure 7, the second tracking detection frame M in the second image acquisition device B is mapped to the image coordinate system of the first image acquisition device A through the homography matrix H to obtain the corresponding mapping frame M′; the fusion cost matrix of IOU and appearance features is calculated based on the first tracking detection frame M and the mapping frame M′ in the first image acquisition device, and the fusion matching is performed using a weighted bipartite graph matching algorithm.

[0171] In one embodiment, the tracking detection box may be represented by two pairs of points.

[0172] Calculate the IOU between all tracking detection frames in the first image coordinate system corresponding to the first image and the mapping frames obtained by mapping all second tracking detection frames to the first image coordinate system, as well as the cosine distance between the appearance features corresponding to the area where the tracking detection frames in the first image are located and the appearance features corresponding to the mapping frames. Calculate the cost matrix according to formula (1), then perform matching association based on the weighted bipartite graph matching algorithm, and modify the tracking identifiers of the two fused target objects into a unified tracking identifier.

[0173] In one embodiment, the appearance feature corresponding to the mapping frame may be an average feature of the target object matched in multiple frames before the current frame.

[0174] This embodiment uses a weighted bipartite graph matching algorithm. Its greatest advantage is that it can achieve optimal matching between the tracking detection frame and the mapping frame. Since the first and second images are known to have overlapping areas, that is, the target object appearing in the overlapping area of ​​the first image acquisition device is likely to also appear in the overlapping area of ​​the second image acquisition device. Therefore, obtaining the maximum number of matching tracking detection frames and mapping frames is more realistic.

[0175] FIG8 is a schematic diagram of a weighted bipartite graph matching algorithm according to an embodiment of the present invention. As shown in FIG8 , uppercase letters are used to represent the target objects in the first image acquisition device, and lowercase letters are used to represent the target objects in the second image acquisition device. The weight of target object A in the first image acquisition device and target object a in the second image acquisition device is 0.45, the weight of target object b is 0.7, and the similarity between target object B in the first image acquisition device and target object b in the second image acquisition device is 0.58. When using maximum similarity matching, due to the high matching probability of the tracking detection box and the mapping box in the overlapping area, the threshold used to determine whether there is a match is difficult to determine. If 0.5 is selected as the threshold, it is obvious that in the matching of FIG8 , A and b will match, while B will not match.

[0176] In this embodiment, a weighted bipartite graph matching algorithm is used. When there is no matching object in B, the association error caused by the maximum similarity matching can be avoided by adjusting the threshold.

[0177] Based on the above embodiment, it can be seen that a target tracking method provided in this embodiment may include:

[0178] Acquire a first image and a second image synchronously captured by a first image capturing device and a second image capturing device, wherein the first image and the second image have an overlapping area;

[0179] Acquire multiple first tracking detection frames obtained by performing target tracking on the first image, and a first tracking identifier corresponding to each first tracking detection frame, and acquire multiple second tracking detection frames obtained by performing target tracking on the second image, and a second tracking identifier corresponding to each second tracking detection frame;

[0180] Mapping the plurality of second tracking detection frames to the first image to obtain corresponding mapping frames according to a homography matrix between the first image acquisition device and the second image acquisition device;

[0181] The multiple first tracking detection frames are matched with the multiple mapping frames using a weighted bipartite graph matching algorithm, and the second tracking identifier of the successfully matched second tracking detection frame is replaced with the first tracking identifier of the first tracking detection frame.

[0182] For each of the multiple tracked targets in the overlapping area of ​​the two image acquisition devices, the weight is calculated based on the IOU and similarity of the tracking detection box, and a weighted bipartite graph matching algorithm is used to fuse the tracking results of the respective trackers of the two image acquisition devices, which can improve the accuracy of tracking detection box fusion when tracking multiple targets with multiple image acquisition devices.

[0183] In the process of mapping the plurality of second tracking detection frames to the first image according to the homography matrix, if the coordinates corresponding to the mapping frames exceed the coordinate range of the first image, the mapping frames exceeding the coordinate range of the first image are deleted.

[0184] The present invention will be described below using a scenario of counting the distribution of people in a target area as an example. When deploying an image acquisition device that cannot cover the entire target area, or when the acquisition effect is poor due to other reasons (such as occlusion), a first image acquisition device and a second image acquisition device can be deployed in the target area. The first image 901 in FIG9A is an image acquired by the first image acquisition device, and the second image 902 is an image acquired by the second image acquisition device. As shown in FIG9A , target tracking is performed on the first image 901 to obtain a tracking identifier of each target (from 1 to 6), and target tracking is performed on the second image 902 to obtain a tracking identifier of each target (from 1001 to 1009).

[0185] When counting the distribution of people in the target area based on the target tracking results of the first image 901 and the target tracking results of the second image 902, it is found that in the overlapping area of ​​the first image 901 and the second image 902, the same target has different tracking identifiers, which will lead to inaccurate statistical results. Therefore, the present invention can be used to fuse the same target in the overlapping area.

[0186] As shown in FIG9B below, the tracking result in the second image 902 is mapped to the first image 901 through the homography matrix. Since the tracking detection frames 1007, 1008, and 1009 in the second image 902 exceed the image range of the first image after mapping, the tracking detection frames 1007, 1008, and 1009 in the second image 902 are filtered out.

[0187] After mapping the tracking result in the second image 902 to the first image 901 through the homography matrix, the IOU cost matrix H and the feature cost matrix F are calculated, and then the relationship between the two is comprehensively considered to generate the cost matrix C, as shown in formula (6).

[0188] In formula (6), γ represents the weight of the appearance feature.

[0189] Based on the cost matrix C, a weighted bipartite graph matching algorithm is used to fuse the first image 901 and the second image 902, thereby ensuring that the same target object has a unique tracking identifier in multi-camera vision. As shown in FIG9C , after fusion, it is determined that the target object with tracking identifier 1 in the first image 901 and the target object with tracking identifier 1006 in the second image 902 are the same target object. Therefore, the tracking identifier of the target object in the second image 902 can be replaced with 1. Similarly, the tracking identifier 1001 in the second image 902 can be replaced with 6, the tracking identifier 1002 in the second image 902 can be replaced with 5, the tracking identifier 1003 in the second image 902 can be replaced with 4, the tracking identifier 1004 in the second image 902 can be replaced with 2, and the tracking identifier 1005 in the second image 902 can be replaced with 3.

[0190] According to the above matching process, target objects in the overlapping area can be fused. In other words, the fusion includes unifying different tracking identifiers of the same target object in the overlapping area, so that the same target object has a globally unique tracking identifier in the target area. In this embodiment, to ensure the accuracy of the fusion, the fusion results can be verified and any incorrect fusion results can be corrected.

[0191] Therefore, after replacing the second tracking identifier of the second tracking detection frame corresponding to the mapping frame with the first tracking identifier of the first tracking detection frame, the method may further include:

[0192] For each image pair other than the synchronized first frame image, obtaining a fusion result of the image pair, the image pair including the synchronized first image and the second image, the fusion result indicating whether a tracking identifier of the same target object in the first image and the second image is consistent;

[0193] If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and one of the tracking identifiers appears for the first time, then the tracking identifier that appears for the first time is replaced with the tracking identifier that has already appeared;

[0194] If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and each tracking identifier does not appear for the first time, the matching similarity between the tracking detection box and the prediction box of the target object in the previous frame image pair is obtained respectively, and the tracking identifier of the target object in the image with the large matching similarity is used to replace the tracking identifier of the target object in the other image.

[0195] FIG10 is a schematic diagram illustrating verification of a fusion result according to an embodiment of the present invention. As shown in FIG10 , if the fusion result indicates that the tracking identifiers of the same target object are consistent, it indicates that the fusion is correct.

[0196] If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, that is, the tracking identifiers of the same target object in the first image and the tracking identifiers in the second image are inconsistent, then it is determined whether the tracker that generated the tracking identifier is the first appearance (i.e., newly initialized). If it is the first appearance, the tracking identifier that appears for the first time is replaced with the tracking identifier that has appeared. The tracking identifier that has appeared can be understood as a tracking identifier that has been correctly tracked multiple times. Considering that the tracking identifier that has appeared has appeared at least twice (the previous frame and the current frame), it means that the probability of stable tracking of the target object is greater than the probability of a new target object appearing. Therefore, if two tracking identifiers appear for the same target object, and one of the tracking identifiers appears for the first time, the tracking identifier that appears for the first time is replaced with the tracking identifier that has appeared.

[0197] For example, target object a is blocked in the field of view of camera 1, but target object a can be stably displayed in the field of view of camera 2. In this case, a tracking identifier has been generated for target object a in the image captured by camera 2. When target object a appears in camera 1, it can be determined through fusion matching that target object a already exists. Therefore, the tracking identifier in camera 2 is used as the globally unique tracking identifier of target object a.

[0198] From the above examples, it can be seen that the problem of tracking mark jumps caused by occlusion of the target object in a single camera can be solved by verifying the fusion results.

[0199] If the fusion result indicates that the tracking identifiers for the same target object are inconsistent, and the trackers that generated the tracking identifiers are not newly created, that is, the target object is not the first time appearing in both image acquisition devices, then this indicates a fusion error. In this case, the similarity of the target object's previous matches in the corresponding image acquisition devices can be determined separately, and the tracking identifier with the higher similarity can be used as the globally unique tracking identifier for the target object.

[0200] For example, the last matching similarity of the target object M in the first image acquisition device is T1, and the last matching similarity in the second image acquisition device is T2. If T1 is greater than T2, the tracking identifier of the target object M in the second image acquisition device is replaced with the tracking identifier in the first image acquisition device.

[0201] When the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and the same target object does not appear for the first time in the two image acquisition devices, it can be determined that a tracking error has occurred in one of the image acquisition devices, such as an exchange of tracking identifiers. In this case, the target object can be determined in which image acquisition device the jump occurred by the tracking matching similarity of the target object in the last single image acquisition device. In principle, it is considered that the image acquisition device with low single target matching similarity has jumped. Therefore, the tracking identifier with high matching similarity is used to correct the tracking identifier with low matching similarity. In the above manner, when the tracking identifier of the target object is exchanged during tracking by a single image acquisition device, the tracking result in the other image acquisition device can be used for correction to improve the tracking accuracy and ensure that the same target object has the same tracking identifier in the two image acquisition devices.

[0202] In this embodiment, when the target is fused, the original tracking identifier is still retained for the tracking result that is not matched with the fusion.

[0203] The present invention uses cameras 1 and 2 with overlapping viewing angles as an example to illustrate the process of implementing target tracking using the present invention. Figure 11 is a schematic diagram of the tracking timing flow according to an embodiment of the present invention. As shown in Figure 11, target tracking is performed on the first frame of synchronization data of camera 1 and camera 2, respectively.

[0204] This example uses target tracking for camera 1 as an example to illustrate the specific tracking process: The first image captured by camera 1 in the first frame of synchronized data is obtained. Target detection is performed on this first image to obtain a tracking detection frame. Target features are extracted from the tracking detection frame. A tracker is created based on the target region, generating a unique tracking ID for each target. As shown in Figure 11, there are four targets in the first image, each with a unique tracking ID of 1, 2, 3, and 4.

[0205] The target tracking process of camera 2 is similar to that of camera 1 and will not be described in detail in this embodiment. As shown in Figure 11, when target tracking is performed on camera 2, it can be found that there are 4 targets in the second image captured by camera 2, and the unique tracking ID of each target is 1000, 1001, 1002 and 1003 respectively.

[0206] All targets in camera 2 are fused with all targets in camera 1 through the homography matrix. That is, the tracking detection boxes corresponding to all targets in camera 2 are mapped to the image coordinate system of camera 1 through the homography matrix. The cost matrix is ​​generated by combining the intersection-over-union ratio of the mapped target area and the target features of the target area, as shown in Formula (1). To obtain the optimal match, this embodiment uses a weighted bipartite graph matching algorithm to achieve matching and merge the matching tracking IDs.

[0207] As shown in Figure 11, the target with tracking ID 2 in camera 1 and the target with tracking ID 1000 in camera 2 are the same target. Therefore, the tracking ID of the target in camera 2 can be replaced from 1000 to 2, that is, the tracking ID of the target in camera 1 is used as the globally unique tracking ID. Similarly, the tracking ID of the target with tracking ID 1002 in camera 2 can be replaced with 4.

[0208] For the Nth (N>1)th frame of synchronization data, similar to the first frame, target tracking is performed on the Nth frame of synchronization data from camera 1 and camera 2, respectively, to obtain the tracking ID of each target in their respective cameras. Theoretically, after fusing the first frame of synchronization data, the same target in the overlapping area will have the same tracking ID in both cameras. Therefore, in subsequent frames, if the fusion result indicates that the tracking IDs of the two matching tracking results are the same, the match is correct; otherwise, correction is required.

[0209] The correction process may include: if the fusion result indicates that the tracking IDs of the two matching tracking results are different, and the target appears for the first time in one of the cameras, then the tracking ID with the higher number of appearances is used as the unified tracking ID for the target. If the target does not appear for the first time in either camera, then the tracking ID with the higher matching confidence level in the single camera is selected as the unified tracking ID for the target.

[0210] Through verification, when the target object is blocked in one of the cameras or the tracking ID of a single camera jumps and causes an error, timely correction can be made to improve the accuracy of target tracking.

[0211] The present invention uses a homography transformation between the images captured by two cameras to achieve coordinate mapping between the two camera images. By combining the target mapping relationship and the similarity of appearance features, the target object is fused in the overlapping area of ​​the cameras, effectively solving the problem of tracking the target object in the critical overlapping area. The present invention also verifies the tracking target in the overlapping area, fully ensuring that the target object in both cameras has a unique tracking identifier, while also effectively addressing the problem of tracking loss caused by single-view occlusion.

[0212] This embodiment can be applied to calculate the distribution of people within the entire target area. For example, Figures 9A, 9B, and 9C show live images of an exhibition area. To calculate the distribution and number of people within the exhibition area, a global population count is required. However, unique tracking identifiers for people in overlapping areas across multiple cameras are a prerequisite for accurate distribution. Therefore, the solution provided by the present invention can be used to fuse people in overlapping areas across multiple cameras, thereby ensuring that the same person has a unique tracking identifier globally.

[0213] This embodiment can be applied to scenarios where the user's area of ​​interest exceeds the capture angle of view of a single camera. For example, when counting the flow of people in a large area, a single camera cannot provide full coverage, and multiple cameras are required to collaboratively analyze and determine. When multiple cameras collaborate to perform personnel analysis, the target fusion method of the present invention can provide effective technical support.

[0214] The tracking results of this embodiment can be displayed on a display screen of a mobile device. The mobile device in this embodiment can be: electronic paper, mobile phone, tablet computer, TV, laptop computer, digital photo frame, navigation device, or any other product or component with a display function.

[0215] This embodiment can correct for occlusions that occur within a single camera. For example, it is known that a single camera may experience severe occlusion, which can severely impact visual analysis results. For example, in the second image 902 of Figure 9A, the individuals tracked with tracking identifiers 1004 and 1005 are severely occluded. In this case, tracking with a single camera may result in failure of individual detection, making individual location and behavior analysis impossible. However, in multi-view camera tracking, unique tracking identifiers can be guaranteed across multiple cameras, allowing effective analysis and determination of target occlusions within a single camera. In other words, when a target object is occluded in one camera, it may not be occluded in the view of another camera. Therefore, the tracking results from the unobstructed camera can be used to ensure accurate individual analysis.

[0216] This embodiment can also be applied to vending machines, where cameras are usually required to monitor changes in items. In order to prevent users from blocking items when taking them, multiple cameras with overlapping acquisition angles can be set up. For example, cameras can be installed on both sides of the top of the vending machine. The solution provided by the present invention can be used to fuse the tracking targets of the two cameras with overlapping areas, which is helpful for analyzing abnormal changes in items in the vending machine.

[0217] In one embodiment, the tracking detection frame can be used as part of the image frame to be displayed and transmitted as a video stream to the display screen of the mobile device for display. This display method is suitable for mobile devices with good CPU and GPU performance.

[0218] In another embodiment, the video stream and the tracking detection frame in each frame of the image determined by the present invention and the tracking identifier corresponding to the tracking detection frame can be transmitted to a mobile device through different data channels. After synchronous calibration by the graphics card of the mobile device, the picture carrying the tracking detection frame and the tracking identifier is displayed on the display screen.

[0219] For example, a mobile device might receive a normal video stream on one data channel, while another receives data packets containing tracking frames with coordinates. The graphics card interprets each frame and, based on the time information associated with the tracking frames in the data packets, combines the tracking frames and tracking markers corresponding to each frame with the image, reproducing the image with the tracking frames on the display.

[0220] FIG12 is a schematic diagram of a target tracking device according to an embodiment of the present invention. As shown in FIG12 , the target tracking device includes:

[0221] An acquisition unit 1201 is configured to acquire a first image and a second image synchronously acquired by a first image acquisition device and a second image acquisition device, wherein the first image and the second image have an overlapping area; and further configured to acquire a plurality of first tracking detection frames and a plurality of second tracking detection frames for performing target tracking on the first image and the second image, respectively;

[0222] A mapping unit 1202 is configured to map the plurality of second tracking detection frames to the first image to obtain corresponding mapping frames according to a mapping relationship between the first image acquisition device and the second image acquisition device;

[0223] The fusion unit 1203 is configured to fuse the target objects in the overlapping area according to the intersection-over-union ratios (IoUs) and similarities of appearance features between the multiple first tracking detection frames and the multiple mapping frames.

[0224] In some embodiments, the fusion unit 1203 is specifically configured to:

[0225] Determining a weight for matching each first tracking detection frame with each mapping frame according to an intersection-over-union ratio between each first tracking detection frame and each mapping frame, and a similarity in appearance features between each first tracking detection frame and each mapping frame;

[0226] Matching the plurality of first tracking detection frames with the plurality of mapping frames using a weighted bipartite graph matching algorithm;

[0227] If the first tracking detection frame matches the mapping frame, it is determined that the target object in the mapping frame and the target object in the first tracking detection frame are the same target object.

[0228] In some embodiments, the fusion unit 1203 is specifically configured to:

[0229] acquiring a first tracking identifier assigned to each first tracking detection frame when performing target tracking on the first image, where the first tracking identifier is globally unique;

[0230] acquiring a second tracking identifier assigned to each second tracking detection frame when performing target tracking on the second image, where the second tracking identifier is globally unique;

[0231] When it is determined that the target object in the mapping frame and the target object in the first tracking detection frame are the same target object, the second tracking identifier of the second tracking detection frame corresponding to the mapping frame is replaced with the first tracking identifier of the first tracking detection frame.

[0232] In some embodiments, the apparatus further comprises a verification unit configured to:

[0233] After replacing the second tracking identifier of the second tracking detection frame corresponding to the mapping frame with the first tracking identification of the first tracking detection frame, obtaining, for other image pairs except the synchronized first frame image, a fusion result of the image pairs, the image pairs including the first image and the second image, the fusion result indicating whether the tracking identifiers of the same target object in the first image and the second image are consistent;

[0234] If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and one of the tracking identifiers appears for the first time, then the tracking identifier that appears for the first time is replaced with the tracking identifier that has already appeared;

[0235] If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and each tracking identifier does not appear for the first time, the matching similarity between the tracking detection box and the prediction box of the target object in the previous frame image pair is obtained respectively, and the tracking identifier of the target object in the image with the large matching similarity is used to replace the tracking identifier of the target object in the other image.

[0236] In some embodiments, the acquiring unit 1201 is specifically configured to:

[0237] Performing target detection on the first frame of the target image to obtain tracking detection frames of multiple target objects, creating a corresponding tracker and a feature library based on the tracking detection frame of each target object, and assigning a tracking identifier to each target object, the feature library being used to store appearance features of the target object, wherein the target image includes the first image and the second image;

[0238] For target images of other frames except the first frame, using each tracker to predict a prediction frame of the target object, and matching based on the intersection-over-union ratio between each prediction frame and each tracked detection frame, and the similarity between the first appearance feature of each target object in the feature library and the second appearance feature of each target object in the other frame target images;

[0239] For a tracking detection frame that matches the prediction frame, determining the tracking identifier of the prediction frame as the tracking identifier of the tracking detection frame in the target image of the other frame, updating a tracker using the position of the tracking detection frame, and updating the first appearance feature using the second appearance feature;

[0240] For a tracking detection frame that does not match the prediction frame, a corresponding tracker is created according to the tracking detection frame, a tracking identifier is assigned to the target object in the tracking detection frame, and the tracker that generates the prediction frame is deleted.

[0241] In some embodiments, the acquiring unit 1201 is further configured to:

[0242] Using a variety of target tracking algorithms to predict the prediction box of the target object;

[0243] Obtain the intersection-over-union ratio between the prediction boxes predicted by various target tracking algorithms and the tracking detection boxes;

[0244] Each predicted frame is matched with each tracking detection frame based on different intersection-over-union ratios and similarities between the first appearance feature in the feature library and the second appearance feature of the tracking detection frame.

[0245] In some embodiments, the acquiring unit 1201 is further configured to:

[0246] When performing target detection on the target image, if a specified target tracking algorithm among the multiple different target tracking algorithms predicts a prediction frame where a target object exists but fails to detect a tracking detection frame of the target object, then the prediction frame of the target object by the specified target tracking algorithm is determined as the tracking detection frame of the target object;

[0247] Obtaining intersection-over-union ratios between prediction boxes predicted using a plurality of different target tracking algorithms and each tracking detection box in the target image;

[0248] Based on different intersection-over-union ratios and similarities between the first appearance feature in the feature library and the second appearance feature of each tracking detection frame, matching is performed between the multiple prediction frames and the multiple tracking detection frames.

[0249] In some embodiments, the apparatus further includes a mapping relationship obtaining unit configured to:

[0250] Acquire a first calibration image and a second calibration image synchronously acquired by the first image acquisition device and the second image acquisition device;

[0251] detecting a first feature point of the size-invariant feature transformation in the first calibration image and a second feature point of the size-invariant feature transformation in the second calibration image;

[0252] Matching a first feature point in the first calibration image with a second feature point in the second calibration image to obtain a feature point pair;

[0253] A mapping relationship between the first image acquisition device and the second image acquisition device is determined according to the homogeneous coordinates of the first feature point in the feature point pair and the homogeneous coordinates of the second feature point in the feature point pair.

[0254] In some embodiments, the mapping relationship obtaining unit is further configured to:

[0255] After matching the first feature point in the first calibration image with the second feature point in the second calibration image to obtain a feature point pair, obtaining a target first feature point constituting the feature point pair in a specified area of ​​the first calibration image;

[0256] In the second calibration image, obtaining target second feature points that match all target first feature points in the designated area;

[0257] Clustering all target second feature points in a set area, wherein the minimum value of the spacing between different set areas is greater than the size of the set area, and the size of the set area is equal to the size of the designated area;

[0258] Obtaining a target setting area with the largest number of target second characteristic points;

[0259] The feature point pairs where all second feature points of the targets in the target setting area are located are updated to the feature point pairs in the first calibration image and the second calibration image.

[0260] In some embodiments, the acquiring unit 1201 is specifically configured to:

[0261] Using a first stream pulling thread to obtain a first data stream collected by a first image acquisition device, and storing the first data stream in a first queue, using a second stream pulling thread to obtain a second data stream collected by a second image acquisition device, and storing the second data stream in a second queue;

[0262] Decoding the first data stream in the first queue using a first decoding thread to obtain a first image, and storing the first image in a third queue; decoding the second data stream in the second queue using a second decoding thread to obtain a second image, and storing the second image in the third queue;

[0263] If the synchronously captured second image or first image is not received for a set period of time after the first image or the second image is received, target tracking is performed on the received first image or the second image, and the third queue is cleared;

[0264] If the second image or the first image captured synchronously is received after a set time has passed since the first image or the second image was received, the first image and the second image are determined to be images captured synchronously by the first image acquisition device and the second image acquisition device, and the third queue is cleared.

[0265] In the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0266] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0267] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A target tracking method, characterized in that: The method comprises: Acquire a first image and a second image synchronously acquired by a first image acquisition device and a second image acquisition device, wherein the first image and the second image have an overlapping area; Acquire a plurality of first tracking detection frames and a plurality of second tracking detection frames for performing target tracking on the first image and the second image respectively; According to a mapping relationship between the first image acquisition device and the second image acquisition device, mapping the plurality of second tracking detection frames to the first image to obtain corresponding mapping frames; According to the intersection-and-union ratios between the multiple first tracking detection frames and the multiple mapping frames and the similarity of the appearance features, the target objects in the overlapping areas are fused.

2. The method according to claim 1, characterized in that The step of fusing the target objects in the overlapping area according to the intersection-over-union ratios and the similarity of the appearance features between the multiple first tracking detection frames and the multiple mapping frames comprises: Determine a weight for matching each first tracking detection frame with each mapping frame according to an intersection-over-union ratio between each first tracking detection frame and each mapping frame, and a similarity of appearance features between each first tracking detection frame and each mapping frame; Matching the plurality of first tracking detection frames with the plurality of mapping frames using a weighted bipartite graph matching algorithm; If the first tracking detection frame matches the mapping frame, it is determined that the target object in the mapping frame and the target object in the first tracking detection frame are the same target object.

3. The method according to claim 2, characterized in that The method further comprises: Acquire a first tracking identifier assigned to each first tracking detection frame when performing target tracking on the first image, where the first tracking identifier is globally unique; Acquire a second tracking identifier assigned to each second tracking detection frame when performing target tracking on the second image, where the second tracking identifier is globally unique; In determining the target object in the mapping frame and the target object in the first tracking detection frame When the two objects are the same target object, the second tracking identifier of the second tracking detection frame corresponding to the mapping frame is replaced by the first tracking identifier of the first tracking detection frame.

4. The method according to claim 3, characterized in that After replacing the second tracking mark of the second tracking detection frame corresponding to the mapping frame with the first tracking mark of the first tracking detection frame, the method further includes: For other image pairs except the synchronized first frame image, obtaining a fusion result of the image pair, the image pair including the first image and the second image, the fusion result indicating whether the tracking identifiers of the same target object in the first image and the second image are consistent; If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and one of the tracking identifiers appears for the first time, then the tracking identifier that appears for the first time is replaced with the tracking identifier that has appeared; If the fusion result indicates that the tracking identifiers of the same target object are inconsistent, and each tracking identifier does not appear for the first time, the matching similarity between the tracking detection box and the prediction box of the target object in the previous frame image pair is obtained respectively, and the tracking identifier of the target object in the image with large matching similarity is used to replace the tracking identifier of the target object in the other image.

5. The method according to claim 1, characterized in that Acquiring a plurality of first tracking detection frames and a plurality of second tracking detection frames for performing target tracking on the first image and the second image respectively includes: Performing target detection on the first frame target image to obtain tracking detection frames of multiple target objects, creating a corresponding tracker and a feature library based on the tracking detection frame of each target object, and assigning a tracking identifier to each target object, wherein the feature library is used to store appearance features of the target object, and the target image includes the first image and the second image; For target images of other frames except the first frame target image, using each tracker to predict the prediction frame of the target object, and matching based on the intersection-over-union ratio between each prediction frame and each tracking detection frame, and the similarity between the first appearance feature of each target object in the feature library and the second appearance feature of each target object in the other frame target images; For a tracking detection frame that matches a prediction frame, the tracking identifier of the prediction frame is determined as The tracking mark of the tracking detection frame in the target image of the other frame, and updating the tracker by using the position of the tracking detection frame, and updating the first appearance feature by using the second appearance feature; For a tracking detection frame that does not match the prediction frame, a corresponding tracker is created according to the tracking detection frame, a tracking identifier is assigned to a target object in the tracking detection frame, and the tracker that generates the prediction frame is deleted.

6. The method according to claim 5, characterized in that The matching based on the intersection-over-union ratio between each prediction frame and each tracking detection frame, and the similarity between the first appearance feature of each target object in the feature library and the second appearance feature of each target object in the target image of the other frame, includes: Using a variety of different target tracking algorithms to predict the prediction box of the target object; Obtain the intersection-over-union ratio between the prediction boxes predicted by various target tracking algorithms and the tracking detection boxes; Based on different intersection-over-union ratios and similarities between the first appearance feature in the feature library and the second appearance feature of the tracking detection frame, each prediction frame is matched with each tracking detection frame.

7. The method according to claim 6, characterized in that The method further comprises: When performing target detection on the target image, if a specified target tracking algorithm among the multiple different target tracking algorithms predicts a prediction frame where a target object exists, but a tracking detection frame of the target object is not detected, then the prediction frame of the target object by the specified target tracking algorithm is determined as the tracking detection frame of the target object; Obtaining intersection-over-union ratios between prediction boxes predicted by using a plurality of different target tracking algorithms and each tracking detection box in the target image; Based on different intersection-over-union ratios and the similarity between the first appearance feature in the feature library and the second appearance feature of each tracking detection frame, matching is performed between the multiple prediction frames and the multiple tracking detection frames.

8. The method according to claim 1, characterized in that The method further comprises: Acquire a first image acquisition device and a first image acquisition device synchronously acquired a first mark a fixed image and a second calibration image; Detecting a first feature point of the size-invariant feature transformation in the first calibration image and a second feature point of the size-invariant feature transformation in the second calibration image; Matching a first feature point in the first calibration image with a second feature point in the second calibration image to obtain a feature point pair; A mapping relationship between the first image acquisition device and the second image acquisition device is determined according to the homogeneous coordinates of the first feature point in the feature point pair and the homogeneous coordinates of the second feature point in the feature point pair.

9. The method according to claim 8, characterized in that After matching the first feature point in the first calibration image with the second feature point in the second calibration image to obtain a feature point pair, the method further includes: In a specified area of ​​the first calibration image, obtaining a target first feature point constituting a feature point pair in the specified area; In the second calibration image, obtaining target second feature points that match all target first feature points in the designated area; Clustering all target second feature points in set areas, wherein the minimum value of the spacing between different set areas is greater than the size of the set area, and the size of the set area is equal to the size of the designated area; Acquire a target setting area having the largest number of target second characteristic points; The feature point pairs where all the second feature points of the targets in the target setting area are located are updated to the feature point pairs in the first calibration image and the second calibration image.

10. The method according to claim 1, characterized in that The step of acquiring the first image and the second image synchronously acquired by the first image acquisition device and the second image acquisition device comprises: Using a first stream pulling thread to obtain a first data stream collected by a first image acquisition device, and storing the first data stream in a first queue, using a second stream pulling thread to obtain a second data stream collected by a second image acquisition device, and storing the second data stream in a second queue; Decoding the first data stream in the first queue using a first decoding thread, to a first image, and storing the first image in a third queue, decoding the second data stream in the second queue by a second decoding thread to obtain a second image, and storing the second image in the third queue; If the second image or the first image acquired synchronously is not received for a set period of time after the first image or the second image is received, target tracking is performed on the received first image or the second image, and the third queue is cleared; If a synchronously captured second image or first image is received after a set time has passed since the first image or the second image was received, the first image and the second image are determined to be images synchronously captured by the first image capture device and the second image capture device, and the third queue is cleared.

11. The method according to claim 1, characterized in that: The method further comprises: According to a mapping relationship between the first image acquisition device and the second image acquisition device, mapping the plurality of first tracking detection frames to the second image to obtain corresponding mapping frames; According to the intersection-and-union ratios between the multiple second tracking detection frames and the multiple mapping frames and the similarity of the appearance features, the target objects in the overlapping areas are fused.

12. A target tracking device, characterized in that: The device comprises: An acquisition unit is used to acquire a first image and a second image synchronously acquired by a first image acquisition device and a second image acquisition device, wherein the first image and the second image have an overlapping area; and is also used to respectively acquire a plurality of first tracking detection frames and a plurality of second tracking detection frames for performing target tracking on the first image and the second image; A mapping unit, configured to map the plurality of second tracking detection frames to the first image to obtain corresponding mapping frames according to a mapping relationship between the first image acquisition device and the second image acquisition device; The fusion unit is used to fuse the target objects in the overlapping area according to the intersection-over-union ratios between the multiple first tracking detection frames and the multiple mapping frames and the similarity of appearance features.

Images