A multi-target real-time trajectory tracking method across view angles

Abstract

The application discloses a cross-view multi-target real-time trajectory tracking method, which combines a machine vision image acquisition system with a master control platform to realize distributed acquisition and centralized processing system architecture, realizes multi-target multi-view real-time image acquisition in a large scene through four sets or more than four sets of large field high-speed cameras, and realizes multi-target motion trajectory tracking. The application uses no less than four sets of acquisition systems to realize full-scene view coverage, solves the problems of poor target positioning accuracy and target position information loss caused by mutual shielding in the single view acquisition mode. Compared with a target positioning method based on a deep learning model, the multi-camera joint solution of the target position can realize higher positioning accuracy, better positioning robustness and higher frame rate.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and computer control, specifically involving distributed sensor networks, multi-target trajectory tracking technology, and real-time video acquisition and processing technology. It is a cross-view multi-target real-time trajectory tracking method. Background Technology

[0002] Patent application CN113283279A discloses a deep learning-based multi-target tracking method in video. This method uses video information of the target in the current environment acquired by a pre-set camera for frame-by-frame processing, separates the target foreground and background information, and inputs them into a target trajectory tracking network to output the predicted motion trajectory of the target. However, target tracking technology based on deep learning methods suffers from problems such as low model accuracy, high computational cost, and poor real-time performance; at the same time, the tracking prediction is easily affected by sudden factors in the actual scene, making it easy to lose track of the target.

[0003] Patent application CN115423846A discloses a multi-target trajectory tracking method and apparatus. It acquires multiple first tracking targets in the current frame and compares them with the image from the previous frame to obtain first predicted bounding boxes corresponding to each target. A Kalman filter algorithm is then used to process the multiple first tracking targets. Compared to deep learning-based target tracking methods, while this method can solve the problems of low tracking efficiency and poor recognition accuracy in multi-target scenarios, it still remains at the level of two-dimensional trajectory tracking technology. It cannot obtain the true coordinates of the target in three-dimensional space over time, thus limiting its practical application.

[0004] Cross-view multi-target real-time trajectory tracking technology can effectively solve the problems of large positioning error and insufficient real-time performance of target trajectory tracking methods based on deep learning models, while reducing the impact of mutual occlusion in multi-target scenarios on invalid target recognition. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a real-time trajectory tracking method for multiple targets across large scenes and perspectives, which solves the problems of limited application scope, low positioning accuracy, and weak real-time performance in existing technologies.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for real-time trajectory tracking of multiple targets across a large scene from multiple perspectives, comprising the following steps.

[0007] S1. Camera intrinsic parameter calibration: Use Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix of each camera for no less than four cameras to reduce the negative impact of lens distortion. At the same time, the accuracy of intrinsic parameter calibration directly affects whether the extrinsic parameter calibration in the next stage can achieve the required positioning accuracy.

[0008] S2. Camera extrinsic parameter calibration: Select, but not limited to, four marker points within the acquisition area. Use a total station to determine the three-dimensional coordinates of the marker points in the spatial coordinate system. Each camera captures these marker points to obtain pixel coordinates. Using the PnP calibration method, the rotation matrix R and translation matrix t in the extrinsic parameters can be obtained by using the two-dimensional pixel coordinates of the marker points to the three-dimensional spatial coordinate point pairs. That is, the camera pose parameters are solved. The three-dimensional coordinates of the origin of the camera coordinate system in the spatial coordinate system are obtained by calculation.

[0009] S3. A monocular camera cannot directly obtain the 3D coordinates of a target in a spatial coordinate system from the pixel coordinates of an image, but according to the formula...

[0010]

[0011] It can obtain the relationship between the three-dimensional coordinates of the target, and set the acquisition site plane as the spatial coordinate system. By adding spatial constraints to the aforementioned camera model, the target's position in the plane can be calculated. The coordinates of the projection point on the plane; therefore, the true coordinates of the target in the spatial coordinate system lie on the line connecting the three-dimensional coordinates of the camera coordinate system origin in the spatial coordinate system and the coordinates of the projection point.

[0012] S4. When different cameras at different viewpoints acquire data on the same target, at least four pairs of camera origin coordinates and projection point coordinates can be obtained. The three-dimensional coordinates of the target are obtained by solving the problem of finding the point with the shortest sum of Euclidean distances from these pairs of points through an iterative method.

[0013] S5. When multiple moving targets exist on the field, the K-means clustering algorithm is used to classify the camera coordinates-projection point coordinates of the same target captured by different cameras. Each camera is paired up, and the line segment connecting the 3D coordinates of the camera's origin in the spatial coordinate system to the projection point coordinates is used as the data source. The point with the shortest Euclidean distance to different line segments between the two cameras is used as the initial point of the clustering space. The number of targets is the number of categories. The clustering algorithm divides the initial points into the number of target classes, with each class tending towards a specific spatial coordinate value. The line segments represented by points in the same class can be considered as the same target captured by different cameras. Finally, these point pairs are used for further data analysis.

[0014] The implementation environment of this method includes:

[0015] The number includes, but is not limited to, four video acquisition units, which are responsible for real-time video acquisition and calculation of the target's position. The projection point is on the plane; at the same time, the camera lenses in the unit are all equipped with infrared filters to reduce the impact of ambient light interference.

[0016] The main control unit consists of two parts: the acquisition unit control end and the information receiving and processing end.

[0017] The human-machine interface of the acquisition unit control terminal, combined with the signal generator, can autonomously select to start or stop the camera acquisition work. The signal generator sends square waves of a certain frequency to each external trigger camera simultaneously through the synchronization signal line, realizing simultaneous and same-frame acquisition, and ensuring that the host processes the data points from each acquisition unit at the same time node.

[0018] The information receiving and processing end consists of a high-performance host, which is responsible for receiving and processing multiple video and point information in real time, and finally feeding back the acquired video and calculated target points to the human-computer interaction interface.

[0019] The distributed information transmission unit consists of multiple video acquisition units and a main control unit. The video acquisition units transmit downsampled video and point coordinate information to the main control unit in real time via Socket communication. The main control unit controls the camera acquisition by transmitting signal pulses through a signal generator via a synchronization signal line.

[0020] We have independently designed a helmet specifically for athletes. In addition to its protective function, the helmet is equipped with an 850nm infrared emitting module and an external battery for power supply. Combined with a data acquisition unit (operating in the infrared band), it can effectively separate targets from the environmental background.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] (1) This invention provides a cross-view multi-target real-time trajectory tracking method. Addressing the shortcomings of current deep learning-based target trajectory tracking methods, such as poor positioning accuracy, low frame rate for multi-target localization processing, and high hardware requirements, this invention changes the target trajectory tracking from the visible light band to the infrared band. From the perspective of information source, it reduces environmental information while enhancing target information, significantly reducing the computing power required for processing images with rich information sources, thus achieving high frame rate, multi-target trajectory tracking under large field-of-view conditions. This technology can currently be used to solve the spatial target localization problem in large-space virtual reality, providing a possibility for dynamic localization in large fields of view.

[0023] (2) This invention provides a cross-view multi-target real-time trajectory tracking method. Multiple video acquisition units are arranged in a large space to form an intersecting field of view, which solves the problem that a single camera cannot solve the three-dimensional coordinates of the target. By acquiring the same target from different perspectives, the actual three-dimensional coordinates of the target can be solved jointly, which effectively overcomes the problem of mutual occlusion in multi-target scenes.

[0024] (3) This invention provides a cross-view, multi-target real-time trajectory tracking method. Hardware-level and software-level frame control technology ensures that the main control platform receives the same frame data transmitted from different acquisition units in real time. The camera acquisition mode is changed to external triggering, which is connected to the signal generator of the main control platform via a synchronization signal line. The signal generator directly controls the acquisition cycle and frame rate of all cameras by emitting fixed-frequency pulses. At the same time, at the software level, an acquisition timestamp is configured for each frame of image and data, which enters the data processing stage after being verified by the main control unit. This technology effectively ensures the accuracy of real-time trajectory tracking for high-speed moving targets.

[0025] (4) This invention provides a cross-view multi-target real-time trajectory tracking method, which distinguishes the same target captured by different acquisition units through the K-means clustering idea. Each camera acquires and calculates the line connecting the camera origin and projection point of different targets. Different cameras match the line in pairs, and the point with the shortest Euclidean distance between the two line segments is recorded as the initial centroid of the cluster space. When the number of targets in the field is known, the clustering algorithm assigns all centroids to the clusters closest to them. Finally, the camera origin-projection point line groups of the same target of different cameras are determined, and the next step of data analysis is carried out. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the layered design of the method architecture provided in Embodiment 1 of the present invention;

[0027] Figure 2 This is a schematic diagram of the large field-of-view acquisition unit arrangement provided in Embodiment 1 of the present invention;

[0028] Figure 3 This is a schematic diagram of the monocular camera target localization theory provided in Embodiment 1 of the present invention;

[0029] Figure 4 This is a schematic diagram of the cross-view multi-camera joint localization theory provided in Embodiment 1 of the present invention;

[0030] Figure 5 This is a schematic diagram illustrating multi-objective classification based on the K-means clustering algorithm provided in Embodiment 1 of the present invention;

[0031] Figure 6 This is a schematic diagram of the hardware framework design for the cross-view multi-target real-time trajectory tracking method provided in Embodiment 1 of the present invention;

[0032] Figure 7 This is a schematic diagram of the software framework design for the cross-view multi-target real-time trajectory tracking method provided in Embodiment 1 of the present invention. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the described embodiments are intended only to facilitate understanding of the invention and are not intended to limit it in any way. The drawings are all in a very simplified form and use non-precise proportions, only to facilitate and clarify the illustration of the embodiments of the invention; the structures shown in the drawings are part of the actual structures. Furthermore, the technical features involved in the various embodiments of the invention described below can be combined with each other as long as they do not conflict with each other. Example

[0034] A cross-view, multi-target real-time trajectory tracking method is proposed, suitable for trajectory tracking and reproduction of high-speed moving targets in large scenes. The hierarchical design of the method architecture is illustrated in the diagram below. Figure 1 As shown, it includes the following steps:

[0035] S1. Layout of cross-view acquisition units in large scenes as follows: Figure 2 As shown, four or more high-speed cameras with wide field of view are arranged at the four corners of the field and the field of view overlaps to cover the entire field. First, the basic principle of the multi-target real-time trajectory tracking method is introduced in detail.

[0036] For monocular cameras, according to the formula of the classic camera model

[0037]

[0038] It can be seen that obtaining relatively accurate camera intrinsic and extrinsic parameter matrices directly determines the accuracy of the conversion from pixel coordinates to world coordinates. The camera intrinsic parameters are obtained using the Zhang Zhengyou calibration method. At least four sampling points are selected within the site, and the spatial coordinates of these sampling points are measured using a total station. The camera captures sampling points to obtain pixel coordinates on the image. The camera extrinsic parameters are solved using the PnP algorithm in visual SLAM. Multiple sets of pixel coordinate-spatial coordinate point pairs of sampling points are input to obtain the camera's rotation matrix R and translation matrix t in the world coordinate system.

[0039] The pixel coordinates of the target in the image are known. The above formula reveals the existence of scale invariants. Since these are unknowns, the true coordinates of the target cannot be determined. Based on the camera model formula, spatial constraints are added to it. To obtain the target relative to the camera The projection point on the plane, and the line connecting the true coordinates of the target in the camera coordinate system origin in the world coordinate system and the projection point, such as... Figure 3 As shown.

[0040] At least four cameras are distributed around the perimeter of the site, with their fields of view overlapping. When the target is captured by all cameras, multiple pairs of camera coordinates and projection point coordinates are calculated. Theoretically, the intersection of the lines connecting these pairs represents the target's true coordinates in the spatial coordinate system, such as... Figure 4 As shown. However, in real-world scenarios, numerous errors can cause spatial line segments to not intersect. Therefore, an iterative method is used to find the point in space with the shortest sum of Euclidean distances to all pairs of points, which is the true coordinate. The specific formula is as follows:

[0041]

[0042] The core of the multi-target recognition problem lies in classifying camera coordinate-projection point pairs of the same target from different cameras. Using the K-means clustering approach, two cameras from different perspectives are paired. The point with the shortest sum of Euclidean distances to the two line segments is designated as the initial centroid of the cluster space and placed within the observation field. The known number of targets is used as the number of clusters (classes). The specific formula is as follows:

[0043] This algorithm assigns all centroids to the nearest cluster, such as... Figure 5 As shown. Upon completion of the task, line pairs clusters formed by different cameras targeting the same target can be obtained and ultimately transmitted to the data processing section.

[0044] S2, Hardware framework design for cross-view multi-target real-time trajectory tracking method, as follows: Figure 6 As shown, the implementation environment of the method consists of no fewer than four video acquisition modules and a main control platform. The video acquisition modules use a large field-of-view high-speed industrial camera paired with a high-performance computer to capture real-time images of the target on the field. Each frame of the image is processed by the high-performance computer, which then sends the downsampled image and solution point data to the main control platform. The main control platform consists of a multi-port high-concurrency parallel computer and a signal generator. The computer can receive video and data information transmitted from no fewer than four acquisition modules, and the signal generator synchronously transmits pulse signals of a certain frequency to all cameras to control their acquisition operation. The video acquisition modules and the main control platform achieve control and data transmission via Ethernet, and the camera acquisition cycle is controlled via a synchronization signal line.

[0045] S3, Software framework design for cross-view multi-target real-time trajectory tracking method, such as... Figure 7As shown, this example uses eight video acquisition units in conjunction with the main control platform. Due to the high real-time requirements of this method, the acquisition units are developed using C++. The main task of each acquisition unit is to acquire images, perform preprocessing such as environmental background noise removal, detect bright spots to obtain the pixel coordinates of multiple targets, and calculate the 3D coordinates of the camera coordinate system origin in the spatial coordinate system, as well as the projection point coordinates of the targets. The downsampled image information and point information are packaged and sent to the main control platform via Socket communication. The main control platform, developed using Python, has functions such as controlling the acquisition units and human-computer interaction. It includes eight image receiving, processing, and display threads, eight point data receiving threads, and one single-frame multi-camera point data processing and display thread. Threads use queues to pass information to ensure frame continuity, and each frame of image and point data carries a timestamp as a checksum to ensure real-time processing of the same frame.

[0046] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-view, multi-target real-time trajectory tracking method, characterized in that: Includes the following steps: S1. Perform intrinsic and extrinsic parameter calibration on no less than four cameras; use Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix of each camera; select no less than four marker points in the acquisition area; use a total station to determine the three-dimensional coordinates of the marker points in the spatial coordinate system; and use the PnP calibration method to obtain the rotation matrix R and translation matrix t in the extrinsic parameters of the camera. S2. Distribute no fewer than four video acquisition units in the venue so that the acquisition range covers the entire venue and any position in the venue is within the overlapping field of view of two or more acquisition units. Each acquisition unit captures images of the target using a high-speed camera and calculates the coordinates of the target's projection point in three-dimensional space; S3. Configure an infrared emitting module with a wavelength of 850nm on the target, and add infrared filters to the lens of all video acquisition units to adjust the working band from visible light to infrared band, so that the target information in the acquired image is separated from the environmental background. S4. Obtain the target image and projection point coordinate information collected by all video acquisition units at the same moment through the main control platform, calculate the true coordinates of the target in three-dimensional space, and generate the target's motion trajectory curve. Data acquisition consistency is ensured through a dual synchronization mechanism at both the hardware and software levels. The hardware-level synchronization triggers all cameras to acquire data simultaneously by generating a square wave signal through a signal generator; the software-level synchronization matches timestamps to each frame of image and point data and verifies the same frame data from different cameras. S5. When there are multiple moving targets, use the K-means clustering algorithm. Data on the same target collected by different cameras are classified and separated; the number of targets being measured is used as the number of cluster categories, and camera coordinate-projection point pairs collected by cameras from different perspectives are paired. The point with the shortest sum of Euclidean distances to the two line segments is used as the initial centroid, and the camera coordinate-projection point pairs corresponding to the same category are passed to the data processing part to realize multi-target trajectory tracking.

2. The cross-view multi-target real-time trajectory tracking method as described in claim 1, characterized in that, It uses a specially customized low-distortion, wide-field-of-view fixed-focus lens and a high-resolution industrial camera.

3. The cross-view multi-target real-time trajectory tracking method as described in claim 1, characterized in that, Based on the formula of the classic camera model Obtain 3D points in target space The relationship between them; The data collection site plane is set as a spatial coordinate system. Plane, add Using the plane as a constraint equation, the coordinates of the target's projection point on the plane in the spatial coordinate system are calculated. According to the PnP algorithm in visual SLAM, the three-dimensional coordinates of the origin of the camera coordinate system in the spatial coordinate system are calculated through multiple sets of pixel coordinates-three-dimensional coordinate point pairs obtained by extrinsic parameter calibration. By using multiple sets of cameras to photograph the same target, multiple sets of spatial lines can be obtained. The coordinates of the point with the shortest Euclidean distance from the multiple sets of lines are obtained by using the iterative idea, which are the three-dimensional point coordinates of the target in the spatial coordinate system.

4. The cross-view multi-target real-time trajectory tracking method as described in claim 1, characterized in that, The camera acquisition mode is externally triggered, meaning that the signal generator on the main control platform generates a square wave of a certain frequency to trigger all cameras on the field to start or stop acquisition simultaneously, achieving hardware-level acquisition time synchronization; at the software level, timestamps are matched for each frame of image and point data, and verification is performed when data from different cameras in the same frame is processed together.

Images