Camera tracking method and apparatus, and storage medium

Abstract

The application provides a camera tracking method and device and a storage medium, and relates to the technical field of target tracking. The method comprises the following steps: collecting images of at least one tracking object in a movement process through at least one camera; extracting feature information of the at least one tracking object from a current frame image collected by the at least one camera; detecting the tracking object from the current frame image collected by the at least one camera based on the feature information of the at least one tracking object, and marking a detection frame of each tracking object in a corresponding current frame image; calculating three-dimensional space coordinate information of the at least one tracking object according to space constraint information of the at least one camera and position information of the detection frame of the at least one tracking object in the image; and determining a movement trajectory of the at least one tracking object according to the three-dimensional space coordinate information of the at least one tracking object and historical trajectory information of the at least one tracking object.

Description

Technical Field

[0001] This application relates to the field of target tracking, and more particularly to a camera tracking method, apparatus, and storage medium. Background Technology

[0002] With the development of computer technology, multi-object tracking has become a trend in the field of computer vision.

[0003] There are two main types of existing methods for tracking multiple targets from multiple cameras. One is based on camera views to track multiple targets from multiple cameras. The other is based on spatiotemporal constraints, which calculates the three-dimensional spatial position of the target using the image coordinates of the target in views from multiple angles, and then performs target trajectory pre-clustering and target trajectory matching to track multiple targets from multiple cameras.

[0004] However, whether multi-camera multi-target tracking is based on camera views or spatiotemporal constraints, both methods suffer from false detections and missed detections due to target occlusion and blurred appearance. Furthermore, they struggle with target re-identification and trajectory matching when there are no overlapping areas in the multiple camera views. Consequently, existing multi-camera multi-target tracking methods exhibit low accuracy in multi-camera multi-target tracking. Summary of the Invention

[0005] This application provides a camera tracking method, apparatus, and storage medium, which can solve the problem of low accuracy in multi-camera, multi-target tracking.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, this application provides a camera tracking method, the method comprising: acquiring images of at least one tracked object during its motion using at least one camera, each camera corresponding to one or more of the at least one tracked object; extracting feature information of at least one tracked object from the current frame image acquired by at least one camera when at least one tracked object enters the shooting area of ​​at least one camera; detecting the tracked object from the current frame image acquired by at least one camera based on the feature information of at least one tracked object, and marking the detection box of each tracked object in its corresponding current frame image when at least one tracked object leaves the shooting area of ​​at least one camera; calculating the three-dimensional spatial coordinate information of at least one tracked object based on the spatial constraint information of at least one camera and the position information of the detection box corresponding to at least one tracked object in the image; and determining the motion trajectory of at least one tracked object based on the three-dimensional spatial coordinate information of at least one tracked object and the historical trajectory information of at least one tracked object.

[0008] Based on the above technical solution, the camera tracking method provided in this application can acquire images of at least one tracked object during its movement using at least one camera. Then, based on the feature information of the at least one tracked object, the object is detected from the current frame image acquired by the at least one camera and marked as a detection box. Next, based on the spatial constraint information of the camera and the position information of the detection box in the image, the three-dimensional spatial coordinate information of the tracked object is calculated. Finally, the motion trajectory of the tracked object is determined using the three-dimensional spatial coordinate information and historical trajectory information of the tracked object. In this way, by detecting images containing the tracked object, the position of the tracked object in the image can be converted to three-dimensional spatial coordinates for tracking, avoiding tracking errors or loss of the tracked object, and improving the tracking accuracy in situations with multiple cameras and multiple tracked objects.

[0009] In a first possible implementation of the first aspect, prior to acquiring images of at least one tracked object during motion using at least one camera, the method further includes: configuring distortion parameters and spatial constraint information of at least one camera, the distortion parameters being used to correct the images acquired by at least one camera.

[0010] In the second possible implementation of the first aspect, the aforementioned spatial constraint information includes at least one of the following: the homography transformation matrix of the camera's main shooting plane; the target entry and exit boundaries of the camera's shooting area; and the spatial topological relationship between the cameras.

[0011] In a third possible implementation of the first aspect, after calculating the three-dimensional spatial coordinate information of at least one tracked object as described above, the method further includes: predicting the next position of at least one tracked object based on the three-dimensional spatial coordinate information and the historical trajectory information of at least one tracked object; matching the predicted next position of the tracked object with a detection box; for a first detection box that does not match the predicted next position of the tracked object, determining whether the tracked object in the first detection box is in the area corresponding to the target entry / exit boundary; if the tracked object is not in the area corresponding to the target entry / exit boundary and the previous position of the tracked object is occluded, using the occluded target position to re-predict the next position of the tracked object; or, if the tracked object is in the area corresponding to the target entry / exit boundary, determining that the tracked object has left the shooting area.

[0012] In the fourth possible implementation of the first aspect, after detecting the tracked object in the current frame image acquired from at least one camera and marking the detection box of each tracked object in its corresponding current frame image, the method further includes: determining the target detection box as a high-scoring detection box if the confidence of the tracked object in the target detection box is greater than or equal to a first threshold; or determining the target detection box as a low-scoring detection box if the confidence of the tracked object in the target detection box is less than the first threshold; after predicting the next position of the tracked object, for high-scoring detection boxes that do not match the predicted next position of the tracked object, re-identifying the tracked object based on the feature information and spatial constraint information of the tracked object corresponding to the high-scoring detection box, and determining the motion trajectory of the tracked object.

[0013] Secondly, this application provides a camera tracking device, comprising: an image acquisition module, a feature extraction module, a detection module, a spatial position calculation module, and a trajectory matching module. The image acquisition module is used to acquire images of at least one tracked object during its movement using at least one camera, each camera corresponding to one or more of the at least one tracked object. The feature extraction module is used to extract feature information of at least one tracked object from the current frame image acquired by the image acquisition module through the at least one camera when at least one tracked object enters the shooting area of ​​at least one camera. The detection module is used to detect the tracked object from the current frame image acquired by the at least one camera based on the feature information of the at least one tracked object extracted by the feature extraction module when at least one tracked object leaves the shooting area of ​​the at least one camera, and to mark the detection box of each tracked object in its corresponding current frame image. The spatial position calculation module is used to calculate the three-dimensional spatial coordinates of at least one tracked object based on the spatial constraint information of at least one camera and the position information of the detection box of at least one tracked object detected by the detection module in the image. The trajectory matching module is used to determine the motion trajectory of at least one tracked object based on the three-dimensional spatial coordinates of at least one tracked object and the historical trajectory information of at least one tracked object.

[0014] In a first possible implementation of the second aspect, the apparatus further includes: a configuration module; the configuration module is configured to configure distortion parameters and spatial constraint information of at least one camera before the image acquisition module acquires images of at least one tracked object during motion via at least one camera, the distortion parameters being used by the image acquisition module to correct the images acquired by at least one camera.

[0015] In a second possible implementation of the second aspect, the aforementioned spatial constraint information includes at least one of the following: the homography transformation matrix of the camera's main shooting plane; the target entry and exit boundaries of the camera's shooting area; and the spatial topological relationship between the cameras.

[0016] In a third possible implementation of the second aspect, the device further includes a prediction module and a determination module. The prediction module is used to predict the next position of at least one tracked object based on the three-dimensional spatial coordinates of the at least one tracked object and its historical trajectory information, after the spatial position calculation module calculates the three-dimensional spatial coordinates of the at least one tracked object. The determination module is further used to match the next position of the tracked object predicted by the prediction module with a detection box. For a first detection box that does not match the predicted next position of the tracked object, it determines whether the tracked object in the first detection box is located in the area corresponding to the target entry / exit boundary. The determination module is further used to, if the tracked object is not located in the area corresponding to the target entry / exit boundary and the previous position of the tracked object is occluded, use the occluded target position to re-predict the next position of the tracked object; or, if the tracked object is located in the area corresponding to the target entry / exit boundary, determine that the tracked object has left the shooting area.

[0017] In a fourth possible implementation of the second aspect, the apparatus further includes: a determining module; the determining module is configured to, after the detection module detects the tracked object in the current frame image acquired from at least one camera and marks the detection box of each tracked object in its corresponding current frame image, determine the target detection box as a high-scoring detection box if the confidence of the tracked object in the target detection box is greater than or equal to a first threshold; or determine the target detection box as a low-scoring detection box if the confidence of the tracked object in the target detection box is less than the first threshold; after predicting the next position of the tracked object, for high-scoring detection boxes that do not match the predicted next position of the tracked object, re-identify the tracked object based on the feature information and spatial constraint information of the tracked object corresponding to the high-scoring detection box, and determine the motion trajectory of the tracked object.

[0018] Thirdly, this application provides a camera tracking device, which includes: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is used to run computer programs or instructions to implement the camera tracking method as described in the first aspect and any possible implementation of the first aspect.

[0019] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a terminal, cause the terminal to perform the camera tracking method as described in the first aspect and any possible implementation thereof.

[0020] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a camera tracking device, cause the camera tracking device to perform the camera tracking method as described in the first aspect and any possible implementation thereof.

[0021] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run computer programs or instructions to implement the camera tracking method as described in the first aspect and any possible implementation thereof.

[0022] Specifically, the chip provided in this application embodiment also includes a memory for storing computer programs or instructions. Attached Figure Description

[0023] Figure 1 One of the flowcharts for a camera tracking method provided in this application embodiment;

[0024] Figure 2 A second flowchart illustrating a camera tracking method provided in this application embodiment;

[0025] Figure 3 One of the schematic diagrams illustrating a camera tracking method provided in this application embodiment;

[0026] Figure 4 A third flowchart illustrating a camera tracking method provided in this application embodiment;

[0027] Figure 5 This is a second schematic diagram illustrating a camera tracking method provided in an embodiment of this application.

[0028] Figure 6 This is a schematic diagram of the structure of a camera tracking device provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of another camera tracking device provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0031] The camera tracking method, apparatus, and storage medium provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0032] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0033] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0034] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0035] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0036] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0037] Multi-object tracking has become one of the trend problems in the field of computer vision. Among them, multi-camera multi-object tracking has become a challenging task because it needs to solve problems such as object detection in crowded scenes, object occlusion, small object detection, object association, differentiation of similar appearances, ID switching and re-identification.

[0038] Existing technologies typically employ methods such as deep learning Transformer network models, Siamese neural networks, Kalman filtering, graph convolutional neural networks, and IoU matching to achieve multi-camera multi-target tracking.

[0039] However, for multi-camera, multi-target tracking methods based on camera views, target occlusion and / or blurred appearance can easily lead to false detection or missed detection, further resulting in incorrect target ID switching and target association. For spatiotemporally constrained multi-camera, multi-target tracking methods, since they calculate the target's 3D spatial position based on image coordinates in multiple view angles and then perform pre-clustering and trajectory matching, they fail to adequately address the problem of target re-identification and trajectory matching across different camera views that do not overlap. Therefore, existing multi-camera, multi-target tracking methods have low accuracy in multi-camera, multi-target tracking.

[0040] To address the low accuracy issue in multi-camera, multi-target tracking in existing technologies, this application provides a camera tracking method. The method includes: acquiring images of at least one tracked object during its motion using at least one camera, with each camera corresponding to one or more of the at least one tracked object; extracting feature information of the at least one tracked object from the current frame image acquired by the at least one camera when the at least one tracked object enters the shooting area of ​​the at least one camera; detecting the tracked object from the current frame image acquired by the at least one camera based on the feature information of the at least one tracked object, and marking the detection box of each tracked object in its corresponding current frame image; calculating the three-dimensional spatial coordinate information of the at least one tracked object based on the spatial constraint information of the at least one camera and the position information of the detection box corresponding to the at least one tracked object in the image; and determining the motion trajectory of the at least one tracked object based on the three-dimensional spatial coordinate information of the at least one tracked object and the historical trajectory information of the at least one tracked object.

[0041] Based on the above technical solution, the camera tracking method provided in this application can acquire images of at least one tracked object during its movement using at least one camera. Then, based on the feature information of the at least one tracked object, the object is detected from the current frame image acquired by the at least one camera and marked as a detection box. Next, based on the spatial constraint information of the camera and the position information of the detection box in the image, the three-dimensional spatial coordinate information of the tracked object is calculated. Finally, the motion trajectory of the tracked object is determined using the three-dimensional spatial coordinate information and historical trajectory information of the tracked object. In this way, by detecting images containing the tracked object, the position of the tracked object in the image can be converted to three-dimensional spatial coordinates for tracking, avoiding tracking errors or loss of the tracked object, and improving the tracking accuracy in situations with multiple cameras and multiple tracked objects.

[0042] The camera tracking method of this application is applied to scenarios in which multiple targets are tracked in real time by deploying multiple cameras.

[0043] like Figure 1 The diagram shows a flowchart of a camera tracking method provided in an embodiment of this application. The method includes the following steps S101 to S105:

[0044] S101. Acquire images of at least one tracked object during its movement using at least one camera.

[0045] Each camera corresponds to one or more of the at least one tracking object mentioned above.

[0046] This application embodiment can track an object using at least one camera, which can form a monitoring network, including: at least one camera at the entrance of the monitoring area that captures the same location from multiple angles; at least one camera deployed at monitoring points in the monitoring area; a server cluster for analysis and processing; a storage device for storing the tracking object's feature information, the tracking object's movement trajectory, historical trajectory information, etc.; a network for connecting the server, storage device, and camera; and related security equipment, etc.

[0047] Optionally, the aforementioned at least one camera can be cameras distributed in various locations within a specific area to form a monitoring network; it can also be multiple cameras on the same device; or it can be multiple cameras in any other form. This application does not impose specific limitations on the embodiments described.

[0048] Optionally, the at least one tracked object can be at least one tracked object that can be captured by the camera within the camera's field of view. For example, the tracked object can be multiple pedestrians within the monitored area.

[0049] In this embodiment, the at least one tracked object is a non-stationary object in motion. That is, the position of the tracked object changes at any time.

[0050] It is understood that each of the at least one camera mentioned above can capture at least one tracked object at a time.

[0051] In this embodiment of the application, when the tracked object enters the camera's shooting area, an image of each tracked object can be captured by at least one of the aforementioned cameras.

[0052] Optionally, combined Figure 1 ,like Figure 2 As shown, before step S101 above, the camera tracking method provided in this application embodiment may further include the following step S106.

[0053] S106. Configure distortion parameters and spatial constraint information for at least one camera. The distortion parameters are used to correct images acquired by at least one camera.

[0054] In this embodiment of the application, before acquiring images, the at least one camera can be initialized, that is, the distortion parameters and spatial constraint information of the at least one camera can be set.

[0055] Optionally, the distortion parameters of the camera may include radial distortion parameters and tangential distortion parameters.

[0056] For example, the camera can be calibrated to obtain radial distortion parameters [k1, k2, k3] and tangential distortion parameters [p1, p2].

[0057] Alternatively, radial distortion correction of the image can be performed using formulas (1) and (2).

[0058]

[0059]

[0060] Where, x corrected The x-coordinate of the distortion point on the imager, y corrected The vertical coordinates represent the position of the distortion point on the imager, k1, k2, and k3 are the radial distortion parameters obtained by the camera through calibration, and r is the distance the imager's optical axis center moves from the edge.

[0061] Alternatively, tangential distortion correction can be performed on the image using formulas (3) and (4).

[0062]

[0063]

[0064] Where p1 and p2 are the tangential distortion parameters obtained by calibrating the camera.

[0065] It is understandable that the distortion parameters of the camera can correct the image distortion caused by camera distortion in order to achieve a better and more realistic display effect, thereby facilitating the subsequent calculation of the three-dimensional spatial position.

[0066] Optionally, images captured by cameras that meet the trajectory matching accuracy requirements can be selected for multi-camera, multi-object tracking. For example, a camera with a pinhole camera model can be used.

[0067] Thus, since the distortion parameters and spatial constraint information of the camera can be configured in advance, the images captured by the camera can be corrected based on the distortion parameters of the camera to obtain more accurate images. This makes it easier to calculate the three-dimensional spatial position information based on the accurate captured images and spatial constraint information.

[0068] Optionally, the aforementioned spatial constraint information may include at least one of the following: the homography transformation matrix of the main plane captured by the camera; the target entry and exit boundaries of the camera's shooting area; and the spatial topological relationship between the cameras.

[0069] In this embodiment, the homography transformation matrix of the main plane captured by the camera can be calculated based on the three-dimensional spatial coordinates of the corresponding points in the image and the image coordinates of the corresponding points, thus mapping the coordinates in the image to the three-dimensional spatial coordinates.

[0070] Optionally, the selection of corresponding points can be manually set or can be a default setting. This application does not impose specific limitations on the embodiments.

[0071] For example, four points uniformly distributed on the image boundary can be selected as the four corresponding points required for calculating the homography transformation matrix.

[0072] For example, the homography transformation matrix of the main plane captured by the camera can be calculated using the geographic coordinates and corresponding image coordinates of four corresponding points in the image. It is necessary to set the image coordinates (x, y, z) of the four corresponding points in the camera. l1 y l1 ), (x l2 y l2 ), (x l3 y l3 ), (x l4 y l4 ) and three-dimensional spatial coordinates (x) r1 y r1 ), (x r2 y r2 ), (x r3 y r3 ), (x r4 y r4 ).

[0073] It should be noted that the homography transformation matrix of the main plane captured by the camera can describe the transformation between two images. By stitching images together, it is possible to convert two-dimensional images into three-dimensional images.

[0074] In this embodiment of the application, the setting of the target entry and exit boundary for each camera shooting area can be: marking objects with specific shapes in the images captured by the camera as the boundary that the tracking object can enter and exit.

[0075] For example, rectangular objects in images captured by a camera can be marked as boundaries that the target can enter or exit, such as doors, elevator areas, etc.

[0076] In this embodiment, the spatial topology between cameras can indicate camera views that overlap or are spatially adjacent to the areas captured by the cameras.

[0077] It is understandable that, based on the spatial topology between cameras, the tracking of a target object can move from one camera view to another. In other words, when a target object disappears from the field of view of one camera, the field of view of the next camera where the target object will appear can be predicted based on the spatial topology between the cameras.

[0078] For example, the connectivity and affinity of targets entering and exiting between areas captured by the camera can be described using a triplet approach, denoted by [C]. m C n [p]. Where C can represent the camera's serial number, and p can represent the camera's serial number. m and C n The affinity between them, p takes values ​​between 0 and 1, where 0 represents C. m Cannot reach C n 1 represents C m Only up to C n .

[0079] Thus, since the homography transformation matrix of the main plane of at least one camera, the target entry and exit boundaries of the camera shooting area, and the spatial topological relationship between the cameras can be preset, the three-dimensional spatial coordinate information of the target can be calculated based on the spatial constraint information of the camera in subsequent calculations, thereby achieving accurate tracking of the target.

[0080] S102. When at least one tracked object enters the shooting area of ​​at least one camera, extract feature information of at least one tracked object from the current frame image captured by at least one camera.

[0081] In this embodiment of the application, the at least one camera can continuously capture images to continuously acquire the current frame image.

[0082] Optionally, at least one tracked object in the current frame image captured by the at least one camera can be detected by an object detection algorithm.

[0083] For example, at least one tracked object can be detected using object detection algorithms such as YOLO.

[0084] In this embodiment of the application, when at least one tracked object enters the shooting area of ​​at least one camera, multi-angle image acquisition can be performed on the at least one tracked object.

[0085] For example, such as Figure 3 As shown, when the tracked object 30 enters the shooting area of ​​at least one camera, multi-angle image acquisition of the tracked object 30 can be performed by cameras 31, 32, 33 and 34.

[0086] Optionally, multi-angle images of each tracked object can be obtained by matching time and spatial location.

[0087] Optionally, feature extraction can be performed on multi-angle images of at least one tracked object.

[0088] Optionally, global features of the tracked object can be extracted, or local features of the tracked object can be extracted. Local features can supplement the total features.

[0089] In this embodiment, the feature map can be divided into multiple regions, including locating local regions and predefined blocks from prior knowledge, extracting features of local regions of interest through an attention mechanism, and using corresponding models to extract and match features of key regions.

[0090] For example, in pedestrian tracking, the face can be one of the key areas.

[0091] Optionally, feature extraction can employ a multi-granularity deep learning model, which is a multi-branch network architecture consisting of a global branch and multiple local branches. In each local branch of the model, the globally merged feature map is divided into different numbers of local regions, including keypoint partitioning and horizontal spatial strip partitioning, and local feature representations are learned independently.

[0092] Optionally, after extracting the feature information of the tracked object, a unique ID can be assigned to the tracked object as its identification information.

[0093] S103. When at least one tracked object leaves the shooting area of ​​at least one camera, based on the feature information of at least one tracked object, detect the tracked object from the current frame image captured by at least one camera, and mark the detection box of each tracked object in its corresponding current frame image.

[0094] In this embodiment of the application, at least one tracking object can be detected from the current frame image captured by at least one camera and marked in the form of a detection box.

[0095] For example, the YOLOv7 object detection model can be used to detect tracked objects in an image, resulting in multiple detection boxes that may include the tracked objects.

[0096] Optionally, the detection frame can be a rectangular detection frame, a circular detection frame, or any other form of detection frame. This application does not impose specific limitations on the embodiments.

[0097] In this embodiment of the application, one tracking object can correspond to one detection box.

[0098] Optionally, a lower confidence threshold (e.g., a confidence level less than a preset value) can be used to detect the tracked object in the current frame image captured by at least one camera, so as to detect as many tracked objects as possible.

[0099] S104. Based on the spatial constraint information of at least one camera and the position information of the detection box corresponding to at least one tracked object in the image, calculate the three-dimensional spatial coordinate information of at least one tracked object.

[0100] In this embodiment of the application, the three-dimensional spatial coordinate information of the at least one tracked object can indicate the position information of the at least one tracked object in real space.

[0101] Optionally, the three-dimensional spatial coordinates of the target can be located using a detection bounding box.

[0102] For example, the three-dimensional spatial coordinates of the target can be located by the center of the bottom edge of the detection frame.

[0103] Optionally, the image coordinates can be converted into three-dimensional spatial coordinates based on the homography transformation matrix of the main plane captured by the camera, using formula (5).

[0104]

[0105] Among them, H 3×3 Let x represent the homography transformation matrix. r y r and z r Represents three-dimensional spatial coordinates, x l y l and z l Represents image coordinates.

[0106] Optionally, in the above three-dimensional spatial coordinates, z r A camera can be used to capture the elevation of the main plane.

[0107] S105. Determine the motion trajectory of at least one tracked object based on the three-dimensional spatial coordinate information of at least one tracked object and the historical trajectory information of at least one tracked object.

[0108] In this embodiment of the application, the historical trajectory information of the at least one tracked object can be the trajectory information that has been determined for the at least one tracked object.

[0109] Optionally, for a tracked object whose motion trajectory has been determined, steps S101 to S105 can be repeated on the next frame image containing the tracked object to obtain the latest motion trajectory information of the tracked object.

[0110] This application provides a camera tracking method that uses at least one camera to capture images of at least one tracked object during its movement. Then, based on the feature information of the tracked object, it is detected from the current frame image captured by the at least one camera and marked as a detection box. Next, based on the spatial constraint information of the camera and the position information of the detection box in the image, the three-dimensional spatial coordinates of the tracked object are calculated. Finally, using the three-dimensional spatial coordinates and historical trajectory information of the tracked object, the motion trajectory of the tracked object is determined. In this way, by detecting images containing the tracked object, the position of the tracked object in the image is transformed into three-dimensional spatial coordinates for tracking, avoiding tracking errors or loss of the tracked object, and improving the tracking accuracy in situations with multiple cameras and multiple tracked objects.

[0111] Optionally, combined Figure 1 ,like Figure 4 As shown, after step S104 above, the camera tracking method provided in this application embodiment may further include steps S107 to S109B as described below.

[0112] S107. Based on the three-dimensional spatial coordinate information of at least one tracked object and the historical trajectory information of at least one tracked object, predict the next position of at least one tracked object.

[0113] Optionally, the next location of the tracked object can be predicted based on the historical trajectory and three-dimensional spatial coordinate information of each tracked object that has been determined.

[0114] For example, the Kalman filter algorithm can be used to predict the new position of the trajectory of each tracked object in the current frame.

[0115] S108. Match the next position of the predicted tracked object with the detection box. For the first detection box that does not match the next position of the predicted tracked object, determine whether the tracked object in the first detection box is in the area corresponding to the target entry / exit boundary.

[0116] In this embodiment of the application, the predicted next position of at least one tracked object can be matched with the detection box to obtain the motion trajectory of at least one tracked object.

[0117] For example, the Hungarian algorithm is used to match the tracking object in the detection box and the tracking object in the predicted location based on the similarity. The similarity is calculated by measuring the accuracy of object detection in a specific dataset (Intersection over Union, IoU). Matches with similarity greater than or equal to a specific threshold are matched to obtain the motion trajectory of at least one tracking object.

[0118] Optionally, for a first detection box that does not match the next position of the predicted tracked object, it can be determined whether the tracked object in the first detection box is in the area corresponding to the target entry / exit boundary, so as to determine the motion trajectory of the tracked object.

[0119] S109. Determine whether the tracked object is in the area corresponding to the target's entry / exit boundary.

[0120] S109A. If the tracked object is not in the area corresponding to the target entry / exit boundary, and the previous position of the tracked object is occluded, the occluded target position is used to re-predict the next position of the tracked object.

[0121] S109B: When the tracked object is in the area corresponding to the target entry / exit boundary, determine that the tracked object has left the shooting area.

[0122] It should be noted that the execution relationship between the above steps S109A and S109B is an "OR" relationship.

[0123] Optionally, the occlusion of the tracked object can be understood as follows: in the current frame image captured by at least one of the cameras, the complete tracked object cannot be detected in the current frame image because at least a part of the tracked object is occluded by other objects.

[0124] For example, such as Figure 5 As shown, when the camera 50 is capturing images, the object being tracked 51 is obscured by the pedestrian 52.

[0125] In this embodiment of the application, if the tracked object is not in the area corresponding to the target entry / exit boundary, and the previous position of the tracked object is occluded, the position of the occluded tracked object can be used to re-predict the next position of the tracked object and perform matching again.

[0126] For example, the position of the occluded tracked object can be used to predict the next position of the tracked object using the Kalman algorithm.

[0127] In this embodiment of the application, if the target is in the area corresponding to the target entry and exit boundary, the tracked object can be marked as leaving the shooting area and the tracked object will no longer be tracked.

[0128] Thus, by predicting the next position of the tracked object based on its 3D spatial coordinates and historical trajectory information, and matching the predicted position with the detection box, and determining whether the tracked object has left the shooting area by checking if it is within the area corresponding to the target's entry / exit boundary when the predicted position does not match the detection box, the system can accurately obtain the motion trajectory of the tracked object and stop tracking it after it leaves the shooting area. This improves the accuracy of tracking multiple objects using multiple cameras.

[0129] Optionally, after step S103 above, the camera tracking method provided in this application embodiment may further include the following steps S110 and S111.

[0130] S110. If the confidence level of the tracked object in the target detection box is greater than or equal to the first threshold, the target detection box is determined as a high-scoring detection box; or if the confidence level of the tracked object in the target detection box is less than the first threshold, the target detection box is determined as a low-scoring detection box.

[0131] In this embodiment of the application, the detected detection boxes can be classified according to the confidence level when detecting the target. Those with a confidence level greater than or equal to a first threshold are identified as high-scoring detection boxes, and those with a confidence level less than the first threshold are identified as low-scoring detection boxes.

[0132] Optionally, the confidence level can be determined by the similarity of the feature information of the detected object and the target, and the area of ​​the detected object that is occluded when detecting the target.

[0133] It is understandable that the similarity of the feature information of the detected object and the target, the smaller the area of ​​the detected object that is occluded, the higher the confidence of the detection box and the higher the detection accuracy.

[0134] S111. After predicting the next position of the tracked object, for high-resolution detection boxes that do not match the predicted next position of the tracked object, the tracked object is re-identified based on the feature information and spatial constraint information of the tracked object corresponding to the high-resolution detection box, and the motion trajectory of the tracked object is determined.

[0135] In this embodiment of the application, for high-resolution detection boxes that fail to match the predicted next position of at least one tracked object, the three-dimensional spatial position is analyzed based on the feature information, three-dimensional spatial coordinate information and spatial constraint information of the corresponding tracked object, and it is determined whether the tracked object is in the region corresponding to the target entry / exit boundary.

[0136] If the high-resolution detection box fails to match the predicted next position of at least one tracked object and is not located in the region corresponding to the target entry / exit boundary, then multi-feature matching of the tracked object is performed. That is, the feature information of the tracked object is retrieved in the region corresponding to the target entry / exit boundary using a multi-granularity deep learning model. The feature information of the detected tracked object is matched with the historical trajectory of the tracked object that does not appear in the shooting area of ​​other cameras. The connectivity and affinity of the spatial topology between multiple cameras are analyzed, and the Hungarian algorithm is used to match based on similarity. The similarity is calculated using the IoU distance to obtain the motion trajectory of the tracked object corresponding to the high-resolution detection box.

[0137] If the high-resolution detection box fails to match the predicted next position of at least one tracked object and is located in the region corresponding to the target entry / exit boundary, then the matching is performed again in the historical target trajectory queue of the at least one tracked object. The Hungarian algorithm is used to match based on similarity, and the similarity is calculated using the IoU distance. If the match is successful, the position information of the high-resolution detection box is recorded as the new position of the tracked object, thereby obtaining the motion trajectory of the tracked object corresponding to the high-resolution detection box.

[0138] Optionally, for the historical trajectory of at least one tracked object that did not match a high-resolution detection box, a low-resolution detection box can be used to match the unmatched historical trajectory again to resume tracking of the tracked object in the low-resolution detection box, thus avoiding the problem of tracked object loss or detection error due to occlusion, motion blur or size change.

[0139] In this embodiment of the application, for the historical trajectory of at least one tracking object that has been matched with the high-resolution detection box, the tracking object in the next frame image can be tracked.

[0140] Thus, since the detection boxes can be divided into high-scoring and low-scoring detection boxes according to their confidence level, and when the high-scoring detection box does not match the predicted next position of the tracked object, the tracked object can be re-identified based on the feature information and spatial constraint information of the tracked object corresponding to the high-scoring detection box. Therefore, the next position of the tracked object can be determined more accurately, thereby improving the accuracy of tracking.

[0141] This application embodiment can divide the camera tracking device into functional modules or functional units according to the above method examples. For example, each function can be divided into a separate functional module or functional unit, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or in software functional modules or functional units. The module or unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0142] like Figure 6 The diagram shown is a schematic representation of a camera tracking device provided in an embodiment of this application. The device includes: an image acquisition module 61, a feature extraction module 62, a detection module 63, a spatial position calculation module 64, and a trajectory matching module 65.

[0143] Image acquisition module 61 is used to acquire images of at least one tracked object during its movement using at least one camera, with each camera corresponding to one or more tracked objects among the at least one tracked object; feature extraction module 62 is used to extract feature information of at least one tracked object from the current frame image acquired by image acquisition module 61 through at least one camera when at least one tracked object enters the shooting area of ​​at least one camera; detection module 63 is used to detect tracked objects from the current frame image acquired by at least one camera based on the feature information of at least one tracked object extracted by feature extraction module 62 when at least one tracked object leaves the shooting area of ​​at least one camera, and to mark the detection box of each tracked object in its corresponding current frame image; spatial position calculation module 64 is used to calculate the three-dimensional spatial coordinates of at least one tracked object based on the spatial constraint information of at least one camera and the position information of the detection box of at least one tracked object detected by detection module 63 in the image; trajectory matching module 65 is used to determine the motion trajectory of at least one tracked object based on the three-dimensional spatial coordinates of at least one tracked object and the historical trajectory information of at least one tracked object.

[0144] In one possible implementation, the above-described apparatus further includes: a configuration module; the configuration module is configured to configure distortion parameters and spatial constraint information of at least one camera before the image acquisition module 61 acquires images of at least one tracked object during its motion using at least one camera, wherein the distortion parameters are used by the image acquisition module 61 to correct the images acquired by at least one camera.

[0145] In one possible implementation, the spatial constraint information includes at least one of the following: the homography transformation matrix of the camera's main shooting plane; the target entry and exit boundaries of the camera's shooting area; and the spatial topological relationship between the cameras.

[0146] In one possible implementation, the above-mentioned device further includes: a prediction module and a determination module; the prediction module is used to predict the next position of at least one tracked object after the spatial position calculation module 64 calculates the three-dimensional spatial coordinate information of at least one tracked object and the historical trajectory information of at least one tracked object.

[0147] The aforementioned determining module is also used to match the next position of the tracked object predicted by the prediction module with the detection box. For a first detection box that does not match the next position of the tracked object, it is determined whether the tracked object in the first detection box is in the area corresponding to the target entry / exit boundary.

[0148] The aforementioned determining module is also used to predict the next position of the tracked object by using the occluded target position when the tracked object is not in the area corresponding to the target entry / exit boundary and the previous position of the tracked object is occluded; or, when the tracked object is in the area corresponding to the target entry / exit boundary, to determine that the tracked object has left the shooting area.

[0149] In one possible implementation, the above apparatus further includes: a determination module; the determination module is configured to: detect the tracking object from the current frame image captured by at least one camera, and mark the detection box of each tracking object in its corresponding current frame image; if the confidence of the tracking object in the target detection box is greater than or equal to a first threshold, determine the target detection box as a high-scoring detection box; or if the confidence of the tracking object in the target detection box is less than the first threshold, determine the target detection box as a low-scoring detection box; after predicting the next position of the tracking object, for high-scoring detection boxes that do not match the predicted next position of the tracking object, perform re-identification of the tracking object based on the feature information and spatial constraint information of the tracking object corresponding to the high-scoring detection box, and determine the motion trajectory of the tracking object.

[0150] When implemented in hardware, the image acquisition module 61, feature extraction module 62, detection module 63, spatial position calculation module 64, and trajectory matching module 65 in this embodiment can be integrated onto the processor. Specific implementation methods are as follows: Figure 7 As shown.

[0151] Figure 7A schematic diagram of another possible structure of the camera tracking device involved in the above embodiments is shown. The camera tracking device includes a processor 302 and a communication interface 303. The processor 302 is used to control and manage the actions of the camera tracking device, for example, executing the steps performed by the image acquisition module 61, feature extraction module 62, detection module 63, spatial position calculation module 64, and trajectory matching module 65, and / or performing other processes of the technology described herein. The communication interface 303 is used to support communication between the camera tracking device and other network entities. The camera tracking device may also include a memory 301 and a bus 304, the memory 301 being used to store the program code and data of the camera tracking device.

[0152] The memory 301 may be a memory in a camera tracking device, and the memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include a combination of the above types of memory.

[0153] The processor 302 described above can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0154] Bus 304 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 304 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0155] Figure 8 This is a schematic diagram of the structure of chip 170 provided in an embodiment of this application. Chip 170 includes one or more (including two) processors 1710 and communication interfaces 1730.

[0156] Optionally, the chip 170 also includes a memory 1740, which may include read-only memory and random access memory, and provides operation instructions and data to the processor 1710. A portion of the memory 1740 may also include non-volatile random access memory (NVRAM).

[0157] In some implementations, memory 1740 stores elements such as execution modules or data structures, or subsets thereof, or extended sets thereof.

[0158] In this embodiment of the application, the corresponding operation is executed by calling the operation instructions stored in the memory 1740 (the operation instructions can be stored in the operating system).

[0159] The processor 1710 described above can implement or execute various exemplary logic blocks, units, and circuits described in conjunction with the disclosure of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, units, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0160] The memory 1740 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include combinations of the above types of memory.

[0161] The Bus 1720 can be an Extended Industry Standard Architecture (EISA) bus, etc. The Bus 1720 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0162] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0163] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the camera tracking method described in the above method embodiments.

[0164] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the camera tracking method in the method flow shown in the above method embodiments.

[0165] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires; portable computer disks; hard disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); registers; hard disks; optical fibers; portable compact disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination thereof; or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0166] Embodiments of the present invention provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the camera tracking method described above.

[0167] Since the camera tracking device, computer-readable storage medium, and computer program product in the embodiments of the present invention can be applied to the above methods, the technical effects they can achieve can also be referred to the above method embodiments. The embodiments of the present invention will not be described again here.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0171] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A camera tracking method, characterized in that, The method includes: Before acquiring images of at least one tracked object during its motion using at least one camera, distortion parameters and spatial constraint information for the at least one camera are configured. The distortion parameters are used to correct the images acquired by the at least one camera. The spatial constraint information includes: the homography transformation matrix of the main plane captured by the camera, the target entry and exit boundaries of the camera's shooting area, and the spatial topological relationship between the cameras. The spatial topological relationship is represented by triples and is used to indicate the overlapping or spatially adjacent camera views between the areas captured by the cameras. Images of at least one tracked object during its movement are captured by at least one camera, with each camera corresponding to one or more of the at least one tracked object; when at least one tracked object enters the shooting area of ​​at least one camera, feature information of at least one tracked object is extracted from the current frame image captured by at least one camera; If the at least one tracked object leaves the shooting area of ​​the at least one camera, based on the feature information of the at least one tracked object, the tracked object is detected from the current frame image captured by the at least one camera, and the detection box of each tracked object in its corresponding current frame image is marked. Based on the spatial constraint information of the at least one camera and the position information of the detection box corresponding to the at least one tracked object in the image, the three-dimensional spatial coordinate information of the at least one tracked object is calculated; the three-dimensional spatial coordinate information of the at least one tracked object is used to indicate the position information of the at least one tracked object in real space. Based on the three-dimensional spatial coordinate information of the at least one tracked object and the historical trajectory information of the at least one tracked object, predict the next position of the at least one tracked object; The predicted next position of the tracked object is matched with the detection box. For a first detection box that does not match the predicted next position of the tracked object, it is determined whether the tracked object in the first detection box is in the area corresponding to the target entry / exit boundary. If the tracked object is not in the area corresponding to the target entry / exit boundary, and the previous position of the tracked object is occluded, the next position of the tracked object is re-predicted using the occluded target position; Alternatively, if the tracked object is in the area corresponding to the target entry / exit boundary, it is determined that the tracked object has left the shooting area; The motion trajectory of the at least one tracked object is determined based on the three-dimensional spatial coordinate information of the at least one tracked object and the historical trajectory information of the at least one tracked object.

2. The method of claim 1, wherein, After detecting the tracked object from the current frame image captured by the at least one camera and marking the detection box of each tracked object in its corresponding current frame image, the method further includes: If the confidence level of the tracked object in the target detection box is greater than or equal to a first threshold, the target detection box is determined as a high-scoring detection box; or if the confidence level of the tracked object in the target detection box is less than the first threshold, the target detection box is determined as a low-scoring detection box. After predicting the next position of the tracked object, for high-resolution detection boxes that do not match the predicted next position of the tracked object, the tracked object is re-identified based on the feature information of the tracked object corresponding to the high-resolution detection box and the spatial constraint information, and the motion trajectory of the tracked object is determined.

3. A camera tracking apparatus, characterized by, The device includes: a configuration module, an image acquisition module, a feature extraction module, a detection module, a spatial position calculation module, a trajectory matching module, a prediction module, and a determination module; The configuration module is used to configure the distortion parameters and spatial constraint information of the at least one camera before the image acquisition module acquires images of at least one tracked object during its motion. The distortion parameters are used by the image acquisition module to correct the images acquired by the at least one camera. The spatial constraint information includes: the homography transformation matrix of the main plane of the camera's image capture, the target entry and exit boundaries of the camera's image capture area, and the spatial topological relationship between the cameras. The spatial topological relationship is represented by triples and is used to indicate the overlapping or spatially adjacent camera views between the areas captured by the cameras. The image acquisition module is used to acquire images of the at least one tracked object during its movement using the at least one camera, with each camera corresponding to one or more of the at least one tracked object. The feature extraction module is used to extract feature information of the at least one tracked object from the current frame image captured by the image acquisition module through the at least one camera when the at least one tracked object enters the shooting area of ​​the at least one camera; The detection module is used to detect the tracking object from the current frame image captured by the at least one camera based on the feature information of the at least one tracking object extracted by the feature extraction module when the at least one tracking object leaves the shooting area of ​​the at least one camera, and to mark the detection box of each tracking object in its corresponding current frame image. The spatial position calculation module is used to calculate the three-dimensional spatial coordinates of the at least one tracked object based on the spatial constraint information of the at least one camera and the position information of the detection box of the at least one tracked object detected by the detection module in the image; the three-dimensional spatial coordinates of the at least one tracked object are used to indicate the position information of the at least one tracked object in real space. The prediction module is used to predict the next position of the at least one tracked object based on the three-dimensional spatial coordinate information of the at least one tracked object and the historical trajectory information of the at least one tracked object; The determining module is used to match the next position of the tracked object predicted by the prediction module with the detection box. For a first detection box that does not match the predicted next position of the tracked object, it is determined whether the tracked object in the first detection box is in the area corresponding to the target entry / exit boundary. The determining module is further configured to, when the tracked object is not in the area corresponding to the target entry / exit boundary and the previous position of the tracked object is occluded, use the occluded target position to re-predict the next position of the tracked object; or, when the tracked object is in the area corresponding to the target entry / exit boundary, determine that the tracked object has left the shooting area. The trajectory matching module is used to determine the motion trajectory of the at least one tracked object based on the three-dimensional spatial coordinate information of the at least one tracked object and the historical trajectory information of the at least one tracked object.

4. The apparatus of claim 3, wherein, The device further includes: a determining module; The determining module is configured to, after the detection module detects the tracking object from the current frame image captured by the at least one camera and marks the detection box of each tracking object in its corresponding current frame image, determine the target detection box as a high-scoring detection box if the confidence of the tracking object in the target detection box is greater than or equal to a first threshold; or determine the target detection box as a low-scoring detection box if the confidence of the tracking object in the target detection box is less than the first threshold. After predicting the next position of the tracked object, for high-resolution detection boxes that do not match the predicted next position of the tracked object, the tracked object is re-identified based on the feature information of the tracked object corresponding to the high-resolution detection box and the spatial constraint information, and the motion trajectory of the tracked object is determined.

5. A camera tracking apparatus, characterized by, include: Processor and communication interface; The communication interface is coupled to the processor, which is used to run computer programs or instructions to implement the camera tracking method as described in any one of claims 1-2.

6. A computer-readable storage medium having stored therein instructions, the computer-readable storage medium comprising: When the computer executes the instruction, the computer performs the camera tracking method as described in any one of claims 1-2.

Images