Multi-camera target tracking method and apparatus based on feature recognition

Abstract

This application discloses a multi-camera target tracking method and apparatus based on feature recognition, relating to the field of computer vision technology. The method includes: acquiring a video stream and performing target tracking to obtain several targets; extracting multimodal features from each target and fusing them to obtain a multimodal fused feature vector, and generating a global trajectory segment; triggering a cross-camera matching event when any target to be matched leaves the current camera's field of view or enters the field of view of another camera, and selecting a candidate target set from the other cameras; executing the cross-camera matching event, matching the optimal matching target with the highest similarity to the target to be matched from the candidate target set of other cameras; associating the target to be matched with the optimal matching target, and stitching them together to form a continuous global motion trajectory of the target to be matched. This solves the problems of poor target tracking performance, insufficient trajectory continuity, and low computational efficiency in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-camera target tracking method and apparatus based on feature recognition. Background Technology

[0002] With the advancement of projects such as Safe City and Smart Transportation, video surveillance systems have evolved from single-point, isolated models to large-scale, networked multi-camera collaborative systems. In such systems, tracking specific targets (such as pedestrians and vehicles) across cameras over long distances and continuously—that is, multi-camera target tracking—has become a core and challenging task.

[0003] Traditional multi-camera target tracking methods primarily rely on single features of the target, such as color histograms or simple motion information. However, in practical applications, these methods face numerous challenges:

[0004] 1) Poor target tracking performance: Factors such as changes in lighting, viewpoint, target pose, and occlusion can cause drastic changes in the appearance of the target, making matching methods that rely solely on appearance features ineffective and resulting in poor target tracking performance.

[0005] 2) Insufficient trajectory continuity: There are non-overlapping fields of view between cameras. There is a time difference and spatial interval between when the target disappears from one camera and when it appears in another camera. It is difficult to directly correlate them, resulting in insufficient trajectory continuity.

[0006] 3) Low computational efficiency: When the number of cameras increases and the number of targets becomes large, the computational load for exhaustive matching on a global scale is enormous, resulting in poor real-time performance. Furthermore, some simplified heuristic matching rules are prone to mismatches and missed matches in complex scenarios, leading to trajectory breaks or identity switching. Summary of the Invention

[0007] This invention provides a multi-camera target tracking method and apparatus based on feature recognition, which solves the problems of poor target tracking effect, insufficient trajectory continuity and low computational efficiency in the prior art.

[0008] In a first aspect, embodiments of the present invention provide a multi-camera target tracking method based on feature recognition, the method comprising:

[0009] Using several cameras, video streams are captured, and target tracking is performed on the video streams of each camera to obtain several targets within the field of view of each camera.

[0010] Extract the multimodal features of each target, fuse the multimodal features to obtain the multimodal fused feature vector of each target, and generate the corresponding global trajectory segment;

[0011] When any target to be matched leaves the current camera's field of view or enters the field of view of another camera, a cross-camera matching event is triggered, and a set of candidate targets is selected from other cameras based on global trajectory segments;

[0012] Perform cross-camera matching events, and match the best matching target with the highest similarity to the target to be matched in the candidate target set of other cameras;

[0013] The target to be matched is associated with the best matching target, and then spliced ​​together to form a continuous global motion trajectory of the target to be matched.

[0014] The technical solution provided in this application has at least the following beneficial effects:

[0015] By fusing motion, appearance, and spatiotemporal multimodal features, a more comprehensive and discriminative target description feature is constructed than that of a single feature. Even under complex conditions such as changes in lighting, partial occlusion, or sudden changes in viewpoint, it can effectively distinguish different targets, significantly reducing the false matching rate and improving target tracking performance. Applying the MOABC algorithm to cross-camera matching, through Circle chaotic initialization, dynamic back-learning, and Levy flight strategy, it effectively balances global exploration and local development capabilities, and can find the globally optimal or near-optimal matching solution from complex candidate sets, significantly improving matching accuracy. Precise cross-camera matching and a reliable global ID management mechanism ensure the consistency of target identity when crossing different camera fields of view, effectively solving the trajectory breakage and identity switching problems common in traditional methods, and generating long-distance, continuous, and complete motion trajectories. The two-stage candidate set screening strategy of "strong spatiotemporal constraints + feature initial screening" greatly reduces the search space of the MOABC algorithm, avoids indiscriminate calculations in a large historical database, ensures real-time performance in multi-camera and multi-target scenarios, and improves computational efficiency.

[0016] In one alternative implementation, several cameras are used to capture video streams, and target tracking is performed on the video streams of each camera to obtain several targets within the field of view of each camera, including:

[0017] Using several cameras, raw video streams are captured, uploaded to a cloud data center, and the OpenCV library is used to connect to the raw video streams of each camera for preprocessing, resulting in several preprocessed video streams.

[0018] Each preprocessed video stream is frame-trimmed to obtain several preprocessed images of consecutive frames, which are then input into the YOLOv8 model for target detection to obtain several original targets in each preprocessed image.

[0019] Using the DeepSORT tracker, target tracking is performed on several original targets in the preprocessed images of consecutive frames to obtain several final targets within the field of view of each camera, and to generate the local tracking ID, detection box, and local trajectory segment for each final target.

[0020] In one optional implementation, multimodal features of each target are extracted and fused to obtain a multimodal fused feature vector for each target, and a corresponding global trajectory segment is generated, including:

[0021] Spatiotemporal calibration is performed on several cameras to obtain the homography matrix of each camera. Based on the homography matrix, coordinate transformation is performed on the local trajectory segment of each target to obtain the corresponding transformed local trajectory segment.

[0022] Based on the detection box, multimodal features of each target are extracted according to the transformed local trajectory fragments, and the multimodal features are fused to obtain the multimodal fused feature vector of each target;

[0023] The multimodal fusion feature vector of each target is added to the corresponding local trajectory segment to obtain the corresponding global trajectory segment.

[0024] In one alternative implementation, the multimodal features include motion features, appearance features, and spatiotemporal features for each target.

[0025] In one optional implementation, when any target to be matched leaves the current camera's field of view or enters the field of view of another camera, a cross-camera matching event is triggered, and a candidate target set is filtered from other cameras based on global trajectory segments, including:

[0026] Using the DeepSORT tracker of the current camera, determine whether the global trajectory segment of any target to be matched exceeds the corresponding field of view boundary. If so, determine that the target to be matched has left the current camera's field of view and create the corresponding query event object.

[0027] Using the YOLOv8 model of other cameras, determine whether a new target to be matched has been detected. If so, determine that the target to be matched has entered the field of view of the new camera and create a corresponding query event object.

[0028] If a query event object exists, a cross-camera matching event is triggered, and candidates are filtered from the historical trajectory database of all other cameras based on the global trajectory fragment of the target to be matched, to obtain a candidate target set.

[0029] In one optional implementation, if a query event object exists, a cross-camera matching event is triggered, and candidates are filtered from the historical trajectory databases of all other cameras based on the global trajectory fragments of the target to be matched, resulting in a candidate target set, including:

[0030] If a query event object exists, a cross-camera matching event is triggered, and the corresponding spatiotemporal window is set;

[0031] Based on the spatiotemporal window of the target to be matched, the historical trajectory databases of all other cameras are filtered under strong spatiotemporal constraints to obtain a preliminary set of candidate trajectories.

[0032] Based on the multimodal fusion feature vector in the global trajectory segment of the target to be matched, the preliminary candidate trajectory set is filtered to obtain the final candidate trajectory set;

[0033] By summarizing the candidate targets corresponding to each candidate global trajectory segment in the final candidate trajectory set, we obtain the candidate target set of the target to be matched in the corresponding other cameras.

[0034] In one alternative implementation, a cross-camera matching event is performed using the MOABC algorithm to find the best matching target with the highest similarity to the target to be matched from the candidate target set of other cameras.

[0035] In one alternative implementation, a cross-camera matching event is performed using the MOABC algorithm to match the optimal target with the highest similarity to the target to be matched from the candidate target sets of other cameras, including:

[0036] The matching problem of cross-camera matching events is transformed into a search optimization problem. Based on the similarity between the target to be matched and the candidate targets in the candidate target sets of other cameras, a multi-objective optimization function is set and used as the fitness function of the MOABC algorithm.

[0037] The candidate matching targets are encoded as vectors of nectar sources in the MOABC algorithm, and the nectar source population parameters and maximum number of iterations of the nectar source algorithm are set.

[0038] Based on the nectar source population parameters, within the search space defined by the candidate target set of other cameras, the initial nectar source population is obtained by initialization using the Circle chaotic mapping sequence; each nectar source in the nectar source population corresponds to a candidate matching target;

[0039] Use the fitness function to obtain the fitness value of each initial nectar source in the initial nectar source population;

[0040] A dynamic reverse mechanism is introduced to perform a neighborhood search of the initial nectar source population to obtain an updated first nectar source population and retain the first local optimal nectar source.

[0041] Use the fitness function to obtain the fitness value of each updated first nectar source in the updated first nectar source population;

[0042] By introducing roulette wheel selection and dynamic reverse mechanism, the updated first nectar source population is subjected to follower bee neighborhood search to obtain the updated second nectar source population, and the second local optimum nectar source is retained.

[0043] Use the fitness function to obtain the fitness value of each updated second nectar source in the updated second nectar source population;

[0044] If any nectar source in the updated second nectar source population has not been updated after several iterations, the scout bee behavior is triggered, and the Levy flight strategy is used to update the nectar source, resulting in an updated third nectar source population.

[0045] Using the fitness function, obtain the fitness value of each updated third nectar source in the updated third nectar source population, and retain the third local optimum nectar source in the updated third nectar source population.

[0046] Select the globally optimal honey source from the first, second, and third locally optimal honey sources;

[0047] When the number of iterations reaches the maximum number of iterations or the fitness function value of the global optimal nectar source meets the requirements, the iterative update of the nectar source population is terminated, and the global optimal nectar source of the current iteration is output as the optimal solution.

[0048] The vector of the optimal solution is decoded to obtain the optimal matching target with the highest similarity to the target to be matched in the candidate target set of other cameras.

[0049] In one optional implementation, the target to be matched is associated with the optimal matching target, and the two are spliced ​​together to form a continuous global motion trajectory of the target to be matched, including:

[0050] In the preset global ID mapping table, check if there is an associated global ID for the local tracking ID of the optimal matching target. If there is, set the target to be matched as the global target of the global ID, and add the local trajectory fragment of the optimal matching target to the record corresponding to the global ID in the global ID mapping table.

[0051] If it does not exist, assign a new global ID to the target to be matched, and add the local trajectory fragment of the best matching target to the record corresponding to the new global ID in the global ID mapping table;

[0052] Extract the historical motion trajectory of the global ID from the historical trajectory database of all other cameras, and stitch together the local trajectory fragments of the target to be matched and the optimal matching target with the historical motion trajectory to form a continuous global motion trajectory of the target to be matched.

[0053] Secondly, embodiments of the present invention provide a multi-camera target tracking device based on feature recognition, used to implement a multi-camera target tracking method, the device comprising:

[0054] The target tracking unit is used to acquire video streams using several cameras and track targets in the video streams of each camera to obtain several targets within the field of view of each camera.

[0055] The feature fusion unit is used to extract the multimodal features of each target, fuse the multimodal features to obtain the multimodal fused feature vector of each target, and generate the corresponding global trajectory segment.

[0056] The candidate target filtering unit is used to trigger a cross-camera matching event when any target to be matched leaves the current camera's field of view or enters the field of view of another camera, and to filter out a set of candidate targets from other cameras based on global trajectory segments;

[0057] The target matching unit is used to perform cross-camera matching events, matching the best matching target with the highest similarity to the target to be matched from the candidate target set of other cameras;

[0058] The target association unit is used to associate the target to be matched with the best matching target and splice them together to form a continuous global motion trajectory of the target to be matched.

[0059] A third aspect of this invention provides an electronic device, which includes:

[0060] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0061] The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0062] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention;

[0064] Figure 2This is a flowchart illustrating the steps of a multi-camera target tracking method based on feature recognition provided in an embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of the functional units of a multi-camera target tracking device based on feature recognition provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0067] The present invention will be further described below with reference to the accompanying drawings.

[0068] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0069] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0070] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0071] like Figure 1As shown, the memory 1005, which serves as a storage medium, may include an operating device, a data storage module, a network communication module, a user interface module, and electronic programs.

[0072] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the feature recognition-based multi-camera target tracking device stored in the memory 1005 through the processor 1001 and executes the feature recognition-based multi-camera target tracking method provided in the embodiment of the present invention.

[0073] Reference Figure 2 The present invention provides a multi-camera target tracking method based on feature recognition, the method comprising:

[0074] S201: Use several cameras to capture video streams, and perform target tracking on the video stream of each camera to obtain several targets within the field of view of each camera;

[0075] S202: Extract the multimodal features of each target, fuse the multimodal features to obtain the multimodal fused feature vector of each target, and generate the corresponding global trajectory segment;

[0076] S203: When any target to be matched leaves the current camera's field of view or enters the field of view of another camera, a cross-camera matching event is triggered, and a set of candidate targets is selected from other cameras based on global trajectory segments;

[0077] S204: Perform a cross-camera matching event, matching the best matching target with the highest similarity to the target to be matched in the candidate target set of other cameras;

[0078] S205: Associate the target to be matched with the optimal matching target and stitch them together to form a continuous global motion trajectory of the target to be matched.

[0079] The technical solution provided in this application has at least the following beneficial effects:

[0080] By fusing motion, appearance, and spatiotemporal multimodal features, a more comprehensive and discriminative target description feature is constructed than that of a single feature. Even under complex conditions such as changes in lighting, partial occlusion, or sudden changes in viewpoint, it can effectively distinguish different targets, significantly reducing the false matching rate and improving target tracking performance. The Multi-Objective Artificial Bee Colony (MOABC) algorithm is applied to cross-camera matching. Through Circle chaotic initialization, dynamic back-learning, and Levy flight strategies, it effectively balances global exploration and local development capabilities, and can find the globally optimal or near-optimal matching solution from complex candidate sets, significantly improving matching accuracy. The accurate cross-camera matching and reliable global ID management mechanism ensure the consistency of the target's identity when crossing different camera fields of view, effectively solving the trajectory breakage and identity switching problems common in traditional methods, and generating long-distance, continuous, and complete motion trajectories. The two-stage candidate set screening strategy of "strong spatiotemporal constraints + feature initial screening" greatly reduces the search space of the MOABC algorithm, avoids indiscriminate calculations in a large historical database, ensures real-time performance in multi-camera and multi-target scenarios, and improves computational efficiency.

[0081] In one alternative implementation, several cameras are used to capture video streams, and target tracking is performed on the video streams of each camera to obtain several targets within the field of view of each camera, including:

[0082] S2011: Use several cameras to capture raw video streams, upload these raw video streams to the cloud data center, and use the OpenCV library to connect to the raw video streams of each camera for preprocessing to obtain several preprocessed video streams.

[0083] In this embodiment, in a typical intersection monitoring scenario, four high-definition network cameras (Cam1, Cam2, Cam3, Cam4) are deployed to cover the four directions of the intersection. All cameras transmit video streams to the cloud data center in real time via a local area network. In the cloud data center, the server uses Python's OpenCV library to connect to and read the video streams from each camera, and preprocesses the video streams, including image denoising and size normalization, to improve the quality of the video streams.

[0084] S2012: Frame segmentation is performed on each preprocessed video stream to obtain several preprocessed images of consecutive frames, which are then input into the YOLOv8 model for target detection to obtain several original targets in each frame of the preprocessed image.

[0085] In this embodiment, taking the video stream of Cam1 as an example, the server captures image frames at a rate of 25 frames per second and inputs each frame into a pre-trained YOLOv8 model. This model is specifically designed for detecting two types of targets: "pedestrians" and "vehicles". YOLOv8 outputs the bounding boxes (detection boxes) and confidence scores of all detected targets in each frame. Post-processing is performed, including: decoding the tensor output by the model into an easy-to-understand format, such as a list containing multiple detection results; discarding all target detection results with a confidence score lower than 0.5; and applying the Non-Maximum Suppression (NMS) algorithm to eliminate redundant detection boxes for the same target.

[0086] S2013: Using the Deep Simple Online and Realtime Tracking (DeepSORT) tracker, several original targets in the preprocessed images of consecutive frames are tracked to obtain several final targets within the field of view of each camera, and the local tracking ID, detection box and local trajectory segment of each final target are generated.

[0087] In this embodiment, the target detection results (detection box position, category, confidence) of the YOLOv8 model are input into the DeepSORT tracker. The DeepSORT tracker combines Kalman filtering to predict target motion and cascaded matching to handle occlusion. It assigns a unique local tracking ID (such as Cam1_P1, Cam1_V1) to each continuously appearing pedestrian or vehicle and records its position in consecutive frames to form a local trajectory segment.

[0088] The DeepSORT tracker integrates a deep learning appearance descriptor, namely the Re-Identification (Re-ID) model, on the basis of Simple Online and Realtime Tracking (SORT). This enables the target to be re-identified after a brief occlusion, greatly improving the robustness of the tracking.

[0089] In one optional implementation, multimodal features of each target are extracted and fused to obtain a multimodal fused feature vector for each target, and a corresponding global trajectory segment is generated, including:

[0090] S2021: Perform spatiotemporal calibration on several cameras to obtain the homography matrix of each camera, and perform coordinate transformation on the local trajectory segment of each target based on the homography matrix to obtain the corresponding transformed local trajectory segment.

[0091] S2022: Based on the detection box, extract the multimodal features of each target according to the transformed local trajectory fragment, and fuse the multimodal features to obtain the multimodal fused feature vector of each target;

[0092] S2023: Add the multimodal fusion feature vector of each target to the corresponding local trajectory segment to obtain the corresponding global trajectory segment.

[0093] In one optional implementation, spatiotemporal calibration is performed on several cameras to obtain the homography matrix of each camera. Based on the homography matrix, coordinate transformation is performed on the local trajectory segment of each target to obtain the corresponding transformed local trajectory segment, including:

[0094] S20211: Define a unified two-dimensional world coordinate system in all public areas covered by cameras (e.g., with the lower left corner of the site as the origin, east as the X-axis, and north as the Y-axis).

[0095] S20212: Calibrate each camera to obtain its intrinsic parameters (focal length, principal point, distortion coefficients) and extrinsic parameters (rotation matrix, translation vector). This is usually done by photographing a checkerboard calibration board and using OpenCV's cv2.calibrateCamera and other functions.

[0096] S20213: Approximate the monitored scene as a plane (such as the ground). In the field of view of each camera, manually select at least 4 points whose coordinates in the world coordinate system are known (e.g., corners of paving stones, road signs, etc.), and use the cv2.findHomography(src_points, dst_points) function to calculate the homography matrix from the image plane to the world coordinate system.

[0097] S20214: For each point in the local trajectory segment of each target, perform coordinate transformation according to the homography matrix corresponding to the camera to obtain the transformed point and store it in a new list to obtain the corresponding transformed local trajectory segment.

[0098] In one alternative implementation, the multimodal features include motion features, appearance features, and spatiotemporal features for each target.

[0099] In one optional implementation, based on the detection box and the transformed local trajectory fragments, multimodal features of each target are extracted, and the multimodal features are fused to obtain a multimodal fused feature vector for each target, including:

[0100] S20221: Based on the local trajectory segments of each target, calculate the target's velocity, acceleration, direction of motion, etc., to obtain the corresponding motion characteristics;

[0101] S20222: Use a pre-trained YOLOv8 model to extract the high-level semantic feature vector of each target in the image within the detection box to obtain the corresponding appearance features;

[0102] S20223: Extract the temporal and positional features from the transformed local trajectory segments of each target to obtain the corresponding spatiotemporal features;

[0103] S20224: Effectively fuse the features of the above three modalities (e.g., through feature concatenation or weighted summation) to generate a multimodal fusion feature vector that can comprehensively describe the target.

[0104] In one optional implementation, when any target to be matched leaves the current camera's field of view or enters the field of view of another camera, a cross-camera matching event is triggered, and a candidate target set is filtered from other cameras based on global trajectory segments, including:

[0105] S2031: Using the DeepSORT tracker of the current camera, determine whether the global trajectory segment of any target to be matched exceeds the corresponding field of view boundary. If so, determine that the target to be matched has left the current camera's field of view and create the corresponding query event object.

[0106] In this embodiment, when the number of consecutive unmatched frames of a global trajectory segment of a target to be matched exceeds a preset maximum threshold (e.g., 70 frames), the DeepSORT tracker will determine that the target has left the field of view and remove it from the active list. This "removal" action is the most direct and reliable trigger signal.

[0107] S2032: Using the YOLOv8 model of other cameras, determine whether a new target to be matched has been detected. If so, determine that the target to be matched has entered the field of view of the new camera and create a corresponding query event object.

[0108] In this embodiment, when the detection box of a target to be matched cannot match any existing trajectory, the YOLOv8 model will create a new trajectory object for it. This "creating a new trajectory" action is the trigger signal.

[0109] S2033: If a query event object exists, a cross-camera matching event is triggered, and candidates are filtered from the historical trajectory database of all other cameras based on the global trajectory fragment of the target to be matched, to obtain a candidate target set.

[0110] In one optional implementation, if a query event object exists, a cross-camera matching event is triggered, and candidates are filtered from the historical trajectory databases of all other cameras based on the global trajectory fragments of the target to be matched, resulting in a candidate target set, including:

[0111] S20331: If a query event object exists, trigger a cross-camera matching event and set the corresponding spatiotemporal window;

[0112] S20332: Based on the spatiotemporal window of the target to be matched, perform strong spatiotemporal constraint filtering on the historical trajectory database of all other cameras to obtain a preliminary set of candidate trajectories;

[0113] In this embodiment, the spatiotemporal window includes a time window and a spatial window:

[0114] Time window: Set a reasonable time difference threshold. ΔT max This threshold should be greater than or equal to the target's position relative to the camera. C The exit of 1 moves to any other camera. C The maximum time required for entry at point 2, for example, if the camera spacing is 100 meters and the maximum human speed is 2 m / s, then ΔT max = 100 / 2 = 50 seconds;

[0115] Spatial window: Set a reasonable spatial distance threshold D max This threshold should be greater than or equal to the physical size of the monitored area, or, more precisely, it can be defined based on the topological relationship between cameras (such as an adjacency matrix), searching for candidates only from adjacent cameras;

[0116] By traversing the historical trajectory databases of other cameras, only those trajectories that appeared within the spatiotemporal window are retained, forming a preliminary set of candidate trajectories.

[0117] S20333: Based on the multimodal fusion feature vector in the global trajectory segment of the target to be matched, the preliminary candidate trajectory set is filtered to obtain the final candidate trajectory set;

[0118] In this embodiment, the similarity (such as cosine similarity) between the multimodal fusion feature vector of the target to be matched and the multimodal fusion feature vector of each candidate target in the preliminary candidate set is calculated. A high threshold is set, and candidate targets with similarity exceeding the threshold are selected to form the final, high-quality candidate target set.

[0119] S20334: Summarize the candidate targets corresponding to each candidate global trajectory segment in the final candidate trajectory set to obtain the candidate target set of the target to be matched in the corresponding other cameras.

[0120] In this embodiment, a two-stage screening strategy is adopted to reduce the computational complexity of subsequent matching.

[0121] In one alternative implementation, a cross-camera matching event is performed using the MOABC algorithm to find the best matching target with the highest similarity to the target to be matched from the candidate target set of other cameras.

[0122] In one alternative implementation, a cross-camera matching event is performed using the MOABC algorithm to match the optimal target with the highest similarity to the target to be matched from the candidate target sets of other cameras, including:

[0123] S2041: The matching problem of cross-camera matching events is transformed into a search optimization problem. Based on the similarity between the target to be matched and the candidate targets in the candidate target sets of other cameras, a multi-objective optimization function is set and used as the fitness function of the MOABC algorithm.

[0124] formula:

[0125]

[0126] In the formula, It is a multi-objective optimization function; The cosine similarity between the multimodal fusion features of the candidate target and the multimodal fusion feature vector of the target to be matched; The spatiotemporal distance between the candidate target and the target to be matched is used; the fitness function is used to evaluate the quality of the nectar source, and its goal is to maximize the similarity between the target to be matched and the selected candidate target, while minimizing the spatiotemporal difference between them. Optimize the function coefficients for the objective function;

[0127]

[0128] In the formula, The fitness function; For individuals The corresponding multi-objective optimization function value; For individual parameters;

[0129] S2042: Encode the candidate matching targets as vectors of nectar sources for the MOABC algorithm, and set the nectar source population parameters and maximum number of iterations for the nectar source algorithm;

[0130] S2043: Based on the nectar source population parameters, within the search space defined by the candidate target set of other cameras, the initial nectar source population is obtained by initializing using the Circle chaotic mapping sequence; each nectar source in the nectar source population corresponds to a candidate matching target;

[0131] The formula is:

[0132]

[0133] In the formula, For the first i An initial source of nectar; For the first i One chaotic variable; These are the upper and lower bounds of the search space; i For individual indicators;

[0134]

[0135] In the formula, For the first i- One chaotic variable; For control parameters (usually taken as follows) =0.5); For the remainder function;

[0136] S2044: Use the fitness function to obtain the fitness value of each initial nectar source in the initial nectar source population;

[0137] S2045: Introducing a dynamic reverse mechanism to perform a neighborhood search for the initial nectar source population, obtaining an updated first nectar source population, and retaining the first locally optimal nectar source.

[0138] The formula is:

[0139]

[0140] In the formula, For the first i An updated solution; for[ [1,1] is a random number; For the first i,k An initial source of nectar; k For individual indicators;

[0141]

[0142] In the formula, For the first i The updated first inverse solution; These represent the maximum and minimum values ​​of the initial nectar source population;

[0143]

[0144] In the formula, For the first i The first dynamic reverse solution is updated; A random number in the range [0,1].

[0145] exist , as well as The solution with the best fitness is selected as the updated first honey source. All updated first honey sources are traversed, and the solution with the best fitness is selected as the first local optimal honey source among the updated first honey source population.

[0146] In this embodiment, the bees are guided to search within the neighborhood of the current nectar source. A dynamic reverse learning strategy is introduced, which generates both the new solution and its reverse solution, and selects the better one as the new nectar source. Dynamic reverse learning can define the reverse point based on the optimal boundary of the current population, which is more exploratory than standard reverse learning and significantly enhances the mining ability of the algorithm.

[0147] S2046: Use the fitness function to obtain the fitness value of each updated first nectar source in the updated first nectar source population;

[0148] S2047: Introducing roulette wheel selection and dynamic reverse mechanism, the updated first nectar source population is searched by following bees to obtain the updated second nectar source population, and the second local optimal nectar source is retained;

[0149] Follower bees select nectar sources to follow based on information (fitness) shared by the lead bee, with a certain probability. The formula is:

[0150]

[0151] In the formula, To follow the bee i The probability of an updated first honey source; For the first i, The fitness value of the first honey source is updated; SN For parameters of nectar source populations; For individual indicators;

[0152] Similar to the lead bee, the selected nectar source is subjected to neighborhood search and dynamic back learning. The solution with the best fitness is selected as the updated second nectar source. All updated second nectar sources are traversed, and the solution with the best fitness is selected as the second local optimal nectar source from the first local optimal nectar source and the updated second nectar source population.

[0153] In this embodiment, the follower bee selects a nectar source to follow by means of a roulette wheel based on the nectar source information (fitness value) shared by the leader bee, and searches within its neighborhood. Similarly, a dynamic reverse mechanism is also introduced in the neighborhood search of the follower bee to generate higher quality candidate solutions and balance the global exploration and local mining of the algorithm.

[0154] S2048: Use the fitness function to obtain the fitness value of each updated second nectar source in the updated second nectar source population;

[0155] S2049: If any nectar source in the updated second nectar source population has not been updated after several iterations, then the scout bee behavior is triggered, and the Levy flight strategy is used to update the nectar source, thus obtaining the updated third nectar source population.

[0156] The formula is:

[0157]

[0158] In the formula, For the first i A newer third honey source; for Levy The random numbers are distributed as follows: for Levy Step length, and ∈[1,2]; The convergence factor; It is the second local optimal nectar source; For the first i An outdated honey source;

[0159]

[0160] In the formula, These are the maximum and minimum values ​​of the convergence factor; This represents the maximum number of iterations. t This represents the current iteration number;

[0161] In this embodiment, if a nectar source is not updated after a preset number of iterations (i.e., it is trapped in a local optimum), the corresponding scout bee is activated. The traditional method is to randomly generate a new nectar source, while this embodiment uses the Levy flight strategy for global search. Levy flight is a random walk with a step size that follows the Levy distribution. It has the characteristics of combining long-distance jumps and short-distance exploration, which can help the algorithm effectively jump out of local optima and explore new and more promising areas.

[0162] S20410: Use the fitness function to obtain the fitness value of each updated third nectar source in the updated third nectar source population, and retain the third local optimum nectar source in the updated third nectar source population.

[0163] S20411: Select the globally optimal honey source from the first, second, and third locally optimal honey sources;

[0164] S20412: When the number of iterations reaches the maximum number of iterations or the fitness function value of the global optimal nectar source meets the requirements, terminate the iterative update of the nectar source population and output the global optimal nectar source of the current iteration as the optimal solution.

[0165] S20412: Decode the vector of the optimal solution to obtain the optimal matching target with the highest similarity to the target to be matched in the candidate target set of other cameras.

[0166] In one optional implementation, the target to be matched is associated with the optimal matching target, and the two are spliced ​​together to form a continuous global motion trajectory of the target to be matched, including:

[0167] S2051: In the preset global ID mapping table, query whether there is an associated global ID for the local tracking ID of the optimal matching target. If there is, set the target to be matched as the global target of the global ID, and add the local trajectory fragment of the optimal matching target to the record corresponding to the global ID in the global ID mapping table.

[0168] In this embodiment, if it has been associated, it means that the target has been tracked before, and the trajectory fragment of the target to be matched is directly spliced ​​with the historical trajectory under the global ID;

[0169] S2052: If it does not exist, assign a new global ID to the target to be matched, and add the local trajectory fragment of the best matching target to the record corresponding to the new global ID in the global ID mapping table;

[0170] In this embodiment, if there is no association, it means that this is a brand new target or the first time it has been captured across cameras. A new global ID is assigned to it, and a new trajectory record is established.

[0171] S2053: Extract the historical motion trajectory of the global ID from the historical trajectory database of all other cameras, and stitch the local trajectory fragments of the target to be matched and the best matching target with the historical motion trajectory to form a continuous global motion trajectory of the target to be matched.

[0172] In this embodiment, the trajectory segment of the target to be matched before leaving the current camera, the trajectory segment of the optimal matching target in the new camera, and the historical trajectory of the global ID (if any) are spliced ​​together in chronological order to form a complete and continuous global motion trajectory.

[0173] This invention also provides a multi-camera target tracking device based on feature recognition, referring to... Figure 3 The diagram shows a functional unit diagram of a multi-camera target tracking device 300 based on feature recognition according to the present invention. The device may include the following units:

[0174] The target tracking unit 301 is used to use several cameras to acquire video streams and perform target tracking on the video streams of each camera to obtain several targets within the field of view of each camera.

[0175] The feature fusion unit 302 is used to extract the multimodal features of each target, fuse the multimodal features to obtain the multimodal fused feature vector of each target, and generate the corresponding global trajectory segment.

[0176] The candidate target filtering unit 303 is used to trigger a cross-camera matching event when any target to be matched leaves the current camera's field of view or enters the field of view of other cameras, and to filter out a set of candidate targets from other cameras based on global trajectory segments.

[0177] The target matching unit 304 is used to perform cross-camera matching events and match the best matching target with the highest similarity to the target to be matched in the candidate target set of other cameras;

[0178] The target association unit 305 is used to associate the target to be matched with the optimal matching target and splice them together to form a continuous global motion trajectory of the target to be matched.

[0179] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0180] Memory, used to store computer programs;

[0181] The processor, when executing the program stored in the memory, implements the feature recognition-based multi-camera target tracking method of the present invention.

[0182] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0183] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0184] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the feature recognition-based multi-camera target tracking method of the present invention.

[0185] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable vehicles (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0189] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0190] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-camera target tracking method based on feature recognition, characterized in that, The method includes: Using several cameras, video streams are captured, and target tracking is performed on the video streams of each camera to obtain several targets within the field of view of each camera. Extract the multimodal features of each target, fuse the multimodal features to obtain the multimodal fused feature vector of each target, and add the multimodal fused feature vector of each target to the corresponding transformed local trajectory segment to obtain the corresponding global trajectory segment; When any target to be matched leaves the current camera's field of view or enters the field of view of another camera, a cross-camera matching event is triggered, and a set of candidate targets is selected from other cameras based on global trajectory segments, including: Using the DeepSORT tracker of the current camera, determine whether the global trajectory segment of any target to be matched exceeds the corresponding field of view boundary. If so, determine that the target to be matched has left the current camera's field of view and create the corresponding query event object. Using the YOLOv8 model of other cameras, determine whether a new target to be matched has been detected. If so, determine that the target to be matched has entered the field of view of the new camera and create a corresponding query event object. If a query event object exists, a cross-camera matching event is triggered, and candidates are filtered from the historical trajectory databases of all other cameras based on the global trajectory fragments of the target to be matched, resulting in a candidate target set, including: If a query event object exists, a cross-camera matching event is triggered, and the corresponding spatiotemporal window is set; Based on the spatiotemporal window of the target to be matched, the historical trajectory databases of all other cameras are filtered under strong spatiotemporal constraints to obtain a preliminary set of candidate trajectories. Based on the multimodal fusion feature vector in the global trajectory segment of the target to be matched, the preliminary candidate trajectory set is filtered to obtain the final candidate trajectory set; By summarizing the candidate targets corresponding to each candidate global trajectory segment in the final candidate trajectory set, we obtain the candidate target set of the target to be matched in the corresponding other cameras; Perform cross-camera matching events using the MOABC algorithm, matching the best matching target with the highest similarity to the target to be matched from the candidate target sets of other cameras, including: The matching problem of cross-camera matching events is transformed into a search optimization problem. Based on the similarity between the target to be matched and the candidate targets in the candidate target sets of other cameras, a multi-objective optimization function is set and used as the fitness function of the MOABC algorithm. formula: In the formula, It is a multi-objective optimization function; The cosine similarity between the multimodal fusion features of the candidate target and the multimodal fusion feature vector of the target to be matched; The spatiotemporal distance between the candidate target and the target to be matched; Optimize the function coefficients for the objective function; In the formula, The fitness function; For individuals The corresponding multi-objective optimization function value; For individual parameters; The target to be matched is associated with the best matching target, and the local trajectory segments of the target to be matched and the best matching target are spliced ​​with the historical motion trajectory to form a continuous global motion trajectory of the target to be matched.

2. The multi-camera target tracking method based on feature recognition according to claim 1, characterized in that, Using several cameras, video streams are captured, and target tracking is performed on the video streams from each camera to obtain several targets within the field of view of each camera, including: Using several cameras, raw video streams are captured, uploaded to a cloud data center, and the OpenCV library is used to connect to the raw video streams of each camera for preprocessing, resulting in several preprocessed video streams. Each preprocessed video stream is frame-trimmed to obtain several preprocessed images of consecutive frames, which are then input into the YOLOv8 model for target detection to obtain several original targets in each preprocessed image. Using the DeepSORT tracker, target tracking is performed on several original targets in the preprocessed images of consecutive frames to obtain several final targets within the field of view of each camera, and to generate the local tracking ID, detection box, and local trajectory segment for each final target.

3. The multi-camera target tracking method based on feature recognition according to claim 2, characterized in that, Multimodal features are extracted for each target, and these features are fused to obtain a multimodal fused feature vector for each target. This multimodal fused feature vector is then added to the corresponding transformed local trajectory segment to obtain the corresponding global trajectory segment, including: Spatiotemporal calibration is performed on several cameras to obtain the homography matrix of each camera. Based on the homography matrix, coordinate transformation is performed on the local trajectory segment of each target to obtain the corresponding transformed local trajectory segment. Based on the detection box, multimodal features of each target are extracted according to the transformed local trajectory fragments, and the multimodal features are fused to obtain the multimodal fused feature vector of each target; The multimodal fusion feature vector of each target is added to the corresponding transformed local trajectory segment to obtain the corresponding global trajectory segment.

4. The multi-camera target tracking method based on feature recognition according to claim 3, characterized in that, The multimodal features include the motion features, appearance features, and spatiotemporal features of each target.

5. The multi-camera target tracking method based on feature recognition according to claim 4, characterized in that, Perform cross-camera matching events, using the MOABC algorithm to find the best matching target with the highest similarity to the target to be matched in the candidate target set of other cameras, including: The candidate matching targets are encoded as vectors of nectar sources in the MOABC algorithm, and the nectar source population parameters and maximum number of iterations of the nectar source algorithm are set. Based on the nectar source population parameters, within the search space defined by the candidate target set of other cameras, the initial nectar source population is obtained by initialization using the Circle chaotic mapping sequence; each nectar source in the nectar source population corresponds to a candidate matching target; Use the fitness function to obtain the fitness value of each initial nectar source in the initial nectar source population; A dynamic reverse mechanism is introduced to perform a neighborhood search of the initial nectar source population to obtain an updated first nectar source population and retain the first local optimal nectar source. Use the fitness function to obtain the fitness value of each updated first nectar source in the updated first nectar source population; By introducing roulette wheel selection and dynamic reverse mechanism, the updated first nectar source population is subjected to follower bee neighborhood search to obtain the updated second nectar source population, and the second local optimum nectar source is retained. Use the fitness function to obtain the fitness value of each updated second nectar source in the updated second nectar source population; If any nectar source in the updated second nectar source population has not been updated after several iterations, the scout bee behavior is triggered, and the Levy flight strategy is used to update the nectar source, resulting in an updated third nectar source population. Using the fitness function, obtain the fitness value of each updated third nectar source in the updated third nectar source population, and retain the third local optimum nectar source in the updated third nectar source population. Select the globally optimal honey source from the first, second, and third locally optimal honey sources; When the number of iterations reaches the maximum number of iterations or the fitness function value of the global optimal nectar source meets the requirements, the iterative update of the nectar source population is terminated, and the global optimal nectar source of the current iteration is output as the optimal solution. The vector of the optimal solution is decoded to obtain the optimal matching target with the highest similarity to the target to be matched in the candidate target set of other cameras.

6. The multi-camera target tracking method based on feature recognition according to claim 5, characterized in that, The target to be matched is associated with the best matching target, and the local trajectory segments of the target to be matched and the best matching target are spliced ​​with the historical motion trajectory to form a continuous global motion trajectory of the target to be matched, including: In the preset global ID mapping table, check if there is an associated global ID for the local tracking ID of the optimal matching target. If there is, set the target to be matched as the global target of the global ID, and add the local trajectory fragment of the optimal matching target to the record corresponding to the global ID in the global ID mapping table. If it does not exist, assign a new global ID to the target to be matched, and add the local trajectory fragment of the best matching target to the record corresponding to the new global ID in the global ID mapping table; Extract the historical motion trajectory of the global ID from the historical trajectory database of all other cameras, and stitch together the local trajectory fragments of the target to be matched and the optimal matching target with the historical motion trajectory to form a continuous global motion trajectory of the target to be matched.

7. A multi-camera target tracking device based on feature recognition, used to implement the multi-camera target tracking method as described in any one of claims 1-6, characterized in that, The device includes: The target tracking unit is used to acquire video streams using several cameras and track targets in the video streams of each camera to obtain several targets within the field of view of each camera. The feature fusion unit is used to extract the multimodal features of each target, fuse the multimodal features to obtain the multimodal fused feature vector of each target, and generate the corresponding global trajectory segment. The candidate target filtering unit is used to trigger a cross-camera matching event when any target to be matched leaves the current camera's field of view or enters the field of view of another camera, and to filter out a set of candidate targets from other cameras based on global trajectory segments; The target matching unit is used to perform cross-camera matching events, matching the best matching target with the highest similarity to the target to be matched from the candidate target set of other cameras; The target association unit is used to associate the target to be matched with the best matching target and splice them together to form a continuous global motion trajectory of the target to be matched.

Images