A target tracking method, device, system and computer readable storage medium

By combining target detection and tracking from multiple cameras with neural networks and multi-target tracking algorithms, the problem of high computational resource requirements in multi-camera target tracking is solved, achieving efficient global target tracking.

CN111145213BActive Publication Date: 2024-11-29CHINA UNIONPAY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN201911258014.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-10
Publication Date
2024-11-29
Estimated Expiration
2039-12-10

AI Technical Summary

Technical Problem

Existing multi-camera-based target tracking methods suffer from bottlenecks in terms of computational and communication resource requirements. As the number of cameras increases, the computational and communication demands increase significantly, leading to low target tracking efficiency.

Method used

By acquiring the current test frames from multiple cameras, target detection and tracking are performed. Using neural network models and multi-target tracking algorithms, such as DeepSORT, combined with the working areas and relationships of the cameras, the global target trajectory is iteratively updated.

Benefits of technology

It reduces the computational resource requirements, enables target tracking based on multiple cameras, and improves the real-time performance and efficiency of global tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111145213B_ABST
    Figure CN111145213B_ABST
Patent Text Reader

Abstract

The present invention provides a target tracking method, device, system and computer-readable storage medium, the method comprising: obtaining the current frames to be detected of multiple cameras set in a monitoring area; performing target detection on the current frames to be detected of each camera in turn, obtaining a detection frame set corresponding to each camera; performing target tracking according to the detection frame set corresponding to each camera, and determining the global target trajectory according to the tracking result. The above method can reduce the computing resources of target tracking based on multiple cameras.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a target tracking method, device, system and computer-readable storage medium. Background Art

[0002] This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] At present, with the popularization of video surveillance technology and the increasing demand for security, target tracking applied in the field of video surveillance has gradually become one of the hot topics in the field of computer vision research. Tracking the moving trajectory of the target object usually requires obtaining the camera's monitoring area image, performing target detection on the image to identify the target, and tracking the identified target object to obtain the complete trajectory of the target object. Due to the complexity of the monitoring scene and the limited field of view of a single camera, in order to achieve global monitoring, multiple cameras may need to cooperate to achieve global coverage of the monitoring area. However, the existing multi-camera-based target tracking method needs to use deep learning methods to analyze images and achieve target tracking. As the number of cameras increases, the computing resource requirements and communication resource requirements increase significantly at the same time, resulting in a technical bottleneck for target tracking. Summary of the invention

[0004] In view of the problems existing in the above-mentioned prior art, a target tracking method, device and computer-readable storage medium are proposed. By using this method, device and computer-readable storage medium, the above-mentioned problems can be solved.

[0005] The present invention provides the following solutions.

[0006] In a first aspect, a target tracking method is provided, comprising: obtaining current frames to be tested of multiple cameras arranged in a monitoring area; performing target detection on the current frames to be tested of each of the multiple cameras in turn to obtain a detection frame set corresponding to each camera; performing target tracking according to the detection frame set corresponding to each camera, and determining a global target trajectory according to the tracking results.

[0007] In some possible implementations, the method further includes: determining a plurality of frame numbers to be tested, and iteratively acquiring current frames to be tested of a plurality of cameras in a time sequence according to the plurality of frame numbers to be tested, thereby iteratively performing target tracking; wherein, an initial global target trajectory is obtained according to an initial frame number to be tested among the plurality of frame numbers to be tested; and an iteratively updated global target trajectory is obtained according to subsequent frame numbers to be tested among the plurality of frame numbers to be tested.

[0008] In some possible implementations, target detection is performed on the current frame to be tested of each camera, including: inputting the current frame to be tested of each camera into a target detection model for target detection; wherein the target detection model is a pedestrian detection model obtained through neural network training.

[0009] In some possible implementations, after obtaining the detection frame set corresponding to each camera, it also includes: performing a projection transformation on the bottom center point of each detection frame in the detection frame set corresponding to each camera according to the framing position of each camera, so as to determine the ground coordinates of each detection frame.

[0010] In some possible implementations, the viewing areas of multiple cameras at least partially overlap, and the method further includes: dividing the working area of ​​each camera in a ground coordinate system according to the viewing area of ​​each camera; wherein the working areas of each camera do not overlap with each other, and if the ground coordinates of any detection frame corresponding to a first camera among the multiple cameras exceed the corresponding working area, then remove any detection frame from the detection frame set of the first camera.

[0011] In some possible implementations, the method further includes: cutting off a non-critical area in a working area of ​​each camera.

[0012] In some possible implementations, tracking is performed according to a detection frame set corresponding to each camera, including: using a multi-target tracking algorithm, and performing multi-target tracking based on the detection frame set corresponding to each camera, and determining local tracking information corresponding to each camera; wherein the parameters used for multi-target tracking are determined based on the historical frames to be tested of each camera.

[0013] In some possible implementations, the multi-target tracking algorithm is a deepsort algorithm.

[0014] In some possible implementations, the method further includes: adding an identity tag to each detection frame according to the local tracking information corresponding to each camera; and determining an iteratively updated global target trajectory based on the identity tag and ground coordinates of each detection frame.

[0015] In some possible implementations, the method further includes: determining an association relationship between multiple cameras based on their working areas; determining a newly added detection frame and a disappeared detection frame in a corresponding working area based on local tracking information of each camera; associating the newly added detection frame and the disappeared detection frame in different working areas based on the association relationship between the multiple cameras to obtain association information; and determining an iteratively updated global target trajectory based on the association information.

[0016] In a second aspect, a target tracking device is provided, including: an acquisition unit, used to acquire current frames to be tested of multiple cameras set in a monitoring area; a detection unit, used to perform target detection on the current frames to be tested of each camera in the multiple cameras in turn, and obtain a detection frame set corresponding to each camera; a tracking unit, used to track the target according to the detection frame set corresponding to each camera, and determine the global target trajectory according to the tracking results.

[0017] In some possible implementations, the method further includes: a frame selection unit, which is used to determine multiple frame numbers to be tested, and iteratively obtain current frames to be tested of multiple cameras in a time sequence according to the multiple frame numbers to be tested, so as to iteratively perform target tracking; wherein, an initial global target trajectory is obtained according to an initial frame number to be tested among the multiple frame numbers to be tested; and an iteratively updated global target trajectory is obtained according to subsequent frame numbers to be tested among the multiple frame numbers to be tested.

[0018] In some possible implementations, the detection unit is further used to: input the current frame to be detected of each camera into a target detection model for target detection; wherein the target detection model is a pedestrian detection model obtained through neural network training.

[0019] In some possible implementations, the detection unit is further used to: after obtaining the detection frame set corresponding to each camera, perform a projection transformation on the bottom center point of each detection frame in the detection frame set corresponding to each camera according to the framing position of each camera, so as to determine the ground coordinates of each detection frame.

[0020] In some possible implementations, the viewing areas of multiple cameras at least partially overlap, and the device is also used to: divide the working area of ​​each camera in the ground coordinate system according to the viewing area of ​​each camera; wherein the working areas of each camera do not overlap with each other, and if the ground coordinates of any detection frame corresponding to the first camera among the multiple cameras exceed the corresponding working area, then remove the any detection frame in the detection frame set of the first camera.

[0021] In some possible implementations, the detection unit is further configured to: cut off a non-critical area in a working area of ​​each camera.

[0022] In some possible implementations, the tracking unit is further used to: adopt a multi-target tracking algorithm, and perform multi-target tracking based on a detection frame set corresponding to each camera, and determine the local tracking information corresponding to each camera; wherein the parameters adopted for multi-target tracking are determined based on the historical frames to be tested of each camera.

[0023] In some possible implementations, the multi-target tracking algorithm is a deepsort algorithm.

[0024] In some possible implementations, the tracking unit is further used to: add an identity to each detection frame according to the local tracking information corresponding to each camera; and determine an iteratively updated global target trajectory based on the identity and ground coordinates of each detection frame.

[0025] In some possible implementations, the tracking unit is further used to: determine an association relationship between multiple cameras based on the working areas of the multiple cameras; determine a newly added detection frame and a disappeared detection frame in the corresponding working area based on the local tracking information of each camera; associate the newly added detection frame and the disappeared detection frame in different working areas based on the association relationship between the multiple cameras to obtain association information; and determine an iteratively updated global target trajectory based on the association information.

[0026] According to a third aspect, a target tracking system is provided, comprising: a plurality of cameras arranged in a monitoring area, and a target tracking device respectively communicatively connected to the plurality of cameras; wherein the target tracking device is configured to execute the method according to the first aspect.

[0027] In a fourth aspect, a target tracking device is provided, comprising: one or more multi-core processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more multi-core processors, the one or more multi-core processors are enabled to implement: obtaining current frames to be tested of multiple cameras set in a monitoring area; performing target detection on the current frames to be tested of each camera in the multiple cameras in turn, and obtaining a detection frame set corresponding to each camera; performing target tracking according to the detection frame set corresponding to each camera, and determining a global target trajectory according to the tracking results.

[0028] In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program, and when the program is executed by a multi-core processor, the multi-core processor executes the method of the first aspect.

[0029] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: In this embodiment, by performing image detection on the current frames to be tested from each camera in turn, and then performing global tracking in the monitoring area based on the detection results corresponding to each camera, global tracking of target objects in multiple surveillance videos can be achieved based on fewer computing resources, and target tracking based on multiple cameras can be achieved based on fewer computing resources.

[0030] It should be understood that the above description is only an overview of the technical solution of the present invention, so that the technical means of the present invention can be more clearly understood and implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described below by way of example. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] By reading the detailed description of the exemplary embodiments below, the advantages and benefits described herein and other advantages and benefits will be apparent to those skilled in the art. The accompanying drawings are only used for the purpose of illustrating exemplary embodiments and are not to be considered as limiting the present invention. Moreover, the same reference numerals are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0032] Figure 1 A schematic diagram of a flow chart of a target tracking method according to an embodiment of the present invention;

[0033] Figure 2 is a ground schematic diagram of a monitoring area according to an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of framing images of multiple cameras according to an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of current frames to be tested of multiple cameras according to an embodiment of the present invention;

[0036] Figure 5 is a schematic diagram of a set of detection frames corresponding to multiple cameras according to an embodiment of the present invention;

[0037] Figure 6 is a schematic diagram of a global target trajectory according to an embodiment of the present invention;

[0038] Figure 7 is a schematic diagram of the structure of a target tracking device according to an embodiment of the present invention;

[0039] Figure 8 is a schematic structural diagram of a target tracking device according to another embodiment of the present invention;

[0040] Fig. 9 FIG. 4 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.

[0041] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] In the present invention, it should be understood that terms such as “including” or “having” are intended to indicate the existence of features, numbers, steps, behaviors, components, parts or a combination thereof disclosed in the specification, and are not intended to exclude the possibility of the existence of one or more other features, numbers, steps, behaviors, components, parts or a combination thereof.

[0044] It should also be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] When tracking active targets in the monitoring area, image detection can be performed on the current frames to be tested from each camera in turn, and then global tracking can be performed in the monitoring area based on the detection results corresponding to each camera. In this way, global tracking of target objects in multiple monitoring videos can be achieved based on fewer computing resources, reducing the demand for computing resources.

[0046] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

[0047] Figure 1 The flowchart of the target tracking method 100 according to the embodiment of the present invention is schematically shown.

[0048] like Figure 1 As shown, the method 100 may include:

[0049] Step S101, obtaining current frames to be tested of multiple cameras set in a monitoring area;

[0050] Specifically, the monitoring area refers to the sum of the viewing areas of multiple cameras, the multiple cameras include at least two cameras, and the viewing areas of the multiple cameras are adjacent to each other or at least partially overlapped, so that the target object to be tracked can move in the monitoring area and then appear in the viewing area of ​​any one or more cameras. Among them, the current frames to be tested of the multiple cameras are respectively extracted from the monitoring videos of the multiple cameras, and the current frames to be tested of each camera have the same acquisition time. Optionally, the target to be tracked in the present disclosure is preferably a pedestrian. Those skilled in the art can understand that the above-mentioned target to be tracked can also be other movable objects, such as animals, vehicles, etc., and the present disclosure does not make specific restrictions on this.

[0051] For example, in complex monitoring scenarios, such as in corridors, large shopping malls, computer rooms, etc., a large number of cameras are usually used to monitor various areas and obtain multi-channel monitoring videos. Figure 2 A schematic monitoring scene is shown, in which a camera 201 and a camera 202 are arranged. Figure 3The above-mentioned camera 201 and camera 202 are shown in FIG. The surveillance video of camera 201 can be parsed into an image frame sequence (A1, A2, ..., A N ), the surveillance video of the camera 202 can be parsed into an image frame sequence (B1, B2, ..., B N ), wherein the above analysis can be performed online in real time or offline. Based on this, the current test frame A of the two cameras can be extracted from the above multiple image frame sequences in sequence. n and B n To perform the target tracking shown in the present disclosure, wherein the value of the subscript n can be n=1, 2, ..., N.

[0052] In some possible embodiments, the method 100 may further include: determining multiple test frame numbers, iteratively acquiring the current test frames of multiple cameras in a time sequence according to the multiple test frame numbers, thereby iteratively performing target tracking; wherein, obtaining the initial global target trajectory according to the initial test frame number in the multiple test frame numbers; obtaining the iteratively updated global target trajectory according to the subsequent test frame numbers in the multiple test frame numbers. This can reduce the amount of calculation and improve the real-time performance of global tracking.

[0053] Specifically, multiple test frame numbers can be determined according to a preset frame acquisition strategy. For example, for a surveillance video with 24 frames per second, the current test frame A can be acquired from the surveillance video of camera 201 and camera 202 every 5 frames. n and B n , where the value of the subscript n can be n=1, 6, 11, ..., and so on. However, other interval frame numbers can also be used, or frame-by-frame detection can also be used, which is not specifically limited in the present disclosure. Based on this, the current test frame A1 and B1 corresponding to the initial test frame sequence number (n=1) can be used to obtain the initial global target trajectory, and the current test frame A1 and B1 corresponding to the subsequent test frame sequence number (n=6, 11, ..., etc.) can be used to obtain the initial global target trajectory. n and B n Perform iterative target tracking to obtain the iteratively updated global target trajectory.

[0054] like Figure 1 As shown, the method 100 may also include:

[0055] Step S102: performing target detection on the current frame to be detected of each camera in turn, to obtain a detection frame set corresponding to each camera;

[0056] In a possible implementation, target detection is performed on the current frame to be tested of each camera, including: inputting the current frame to be tested of each camera into a target detection model for target detection; wherein the target detection model is a pedestrian detection model obtained through neural network training.

[0057] For example, Figure 4 As shown, the current frame A to be tested of camera 201 and camera 202 is shown. n and B n Then, the preprocessed current frame A is input into any deep learning-based pedestrian detection model. n and B n Perform detection and output a series of pedestrian detection frames for each camera. The purpose of obtaining pedestrian detection frames is to obtain the current frame A to be tested. n and B n , the location information and size information of all pedestrians. The pedestrian detection model can be, for example, a YOLO (Unified Real-time Object Detection, YouOnly Look Once) model, etc., and the present disclosure does not make any specific restrictions on this. Figure 5 As shown, it shows a plurality of current frames A to be tested. n and B n , multiple detection frame sets are obtained by detection, including the detection frame set (a1, a2, a3) corresponding to camera 201 and the detection frame set (b) corresponding to camera 202.

[0058] In a possible implementation, after obtaining the detection frame set corresponding to each camera, the method further includes: performing a projection transformation according to the framing position of each camera and the center point of the bottom of each detection frame in the detection frame set corresponding to each camera, thereby determining the ground coordinates of each detection frame in the detection frame set corresponding to each camera. In this way, the objects recognized within the framing range of each camera can be combined into a unified coordinate system.

[0059] For example, you can get Figure 5 The bottom center point position of each detection frame corresponding to each camera in the image is converted to obtain the actual ground position of the target object in the monitoring scene. Figure 6The figure shows the ground coordinates of each detection frame obtained by projection transformation. Specifically, it can be seen that the ground corridor under the perspective of each camera is an approximately trapezoidal area. Therefore, for the detection frame set corresponding to each camera, the coordinates of the center point of the bottom of each detection frame in the standard rectangular area can be obtained by trapezoid-rectangle transformation. Secondly, the standard rectangular area is rotated according to the actual layout of the monitoring scene, and the rotated coordinates of the center point of the bottom of each detection frame are obtained by rotation matrix calculation. Finally, the rotated coordinates are translated and scaled according to the actual layout of the monitoring scene to obtain the final coordinate position.

[0060] In one possible implementation, the framing areas of multiple cameras at least partially overlap, and the method further includes: dividing the working area of ​​each camera in a ground coordinate system according to the framing area of ​​each camera; wherein the working areas of each camera do not overlap with each other, and if the ground coordinates of any detection frame corresponding to a first camera among the multiple cameras exceed the corresponding working area, then remove any detection frame from the detection frame set of the first camera.

[0061] For example, Figure 2 As shown, in order to eliminate the monitoring blind spots in the monitoring scene, the viewing areas of cameras 201 and 202 actually overlap. Based on this, in order to effectively avoid the problem of coordinate display conflicts, the working area of ​​each camera can be divided. For example, the working area of ​​camera 201 is area X, and the working area of ​​camera 202 is area Y, so that the working areas of each camera are adjacent. Furthermore, the ground coordinates of each detection frame corresponding to each camera must be within the working area of ​​the camera. If they are not within the working area responsible for the camera, they are removed. For example, since the ground coordinates of detection frame a3 in the detection frame set (a1, a2, a3) corresponding to camera 201 are outside the X area, the detection frame a3 is removed from the detection frame set corresponding to camera 201 to obtain (a1, a2) for subsequent operations.

[0062] In one possible implementation, the method further includes: cutting off a non-critical area in the working area of ​​each camera. Specifically, whether it is a critical area can be determined based on the specific layout of the monitoring scene. For example, a ceiling area that pedestrians cannot pass through can be directly cut off, which can reduce the amount of calculation for target tracking.

[0063] like Figure 1 As shown, the method 100 may also include:

[0064] Step S103: Track the target according to the detection frame set corresponding to each camera, and update the global target trajectory according to the tracking result.

[0065] Specifically, as described above, for each camera, target detection can be performed based on the initial current test frames A1 and B1 to determine the initial global target trajectory. n and B n Perform target detection and iteratively track the target based on the target detection results, thereby iteratively updating the global target trajectory.

[0066] In one possible implementation, tracking is performed based on the detection frame set corresponding to each camera, including: using a multi-target tracking algorithm, and performing multi-target tracking based on the detection frame set corresponding to each camera, and determining the local tracking information corresponding to each camera; wherein the parameters used for multi-target tracking are determined based on the historical frames to be tested of each camera. In this way, multi-target tracking can be achieved in the monitoring area.

[0067] Specifically, the multi-target tracking algorithm is a target tracking algorithm based on a single camera, such as the DeepSORT algorithm (Simple Online and Realtime Tracking with a Deep Association Metric), so the local tracking information of each camera can be obtained. Among them, the parameters used in multi-target tracking are determined based on the historical frames to be tested of each camera. Specifically, the target frame to be tracked can be determined when any target first appears in the working area of ​​a camera, and the subsequent frames to be tested of the camera can be tracked based on the multi-target detection algorithm and the target frame with annotated identity to determine the local tracking information of the target in the working area of ​​the camera.

[0068] In a possible implementation, the multi-target tracking algorithm is a deepsort algorithm. Of course, other target tracking algorithms may also be used, and those skilled in the art will appreciate that the present disclosure does not emphasize which specific target tracking algorithm is used.

[0069] In a possible implementation, updating the global target trajectory according to the tracking results also includes: adding an identity tag to each detection frame according to the local tracking information corresponding to each camera; and updating the global target trajectory using the ground coordinates of each detection frame based on the identity tag.

[0070] For example, Figure 6 As shown, the curve part shows the current global target trajectory, that is, the global target trajectory determined in the last iteration process, and the points a1, a2 and b respectively represent Figure 5The ground coordinates of multiple detection frames are shown in . Among them, if the local tracking information corresponding to the camera 201 indicates that the detection frame a2 matches the existing "target 2" feature, the detection frame a2 is labeled "target 2" and the ground coordinates of point a2 are added to the existing track of "target 2" (i.e. Figure 6 If the local tracking information corresponding to the camera 201 indicates that there is no matching target at the detection frame point a1, a label “target 3” is added to the detection frame a1, and a new track of “target 3” is created.

[0071] In a possible implementation, updating the global target trajectory according to the tracking results also includes: determining the association relationship between the multiple cameras based on the working areas of the multiple cameras; determining the newly added detection frames and the disappeared detection frames in the corresponding working area based on the local tracking information of each camera; associating the newly added detection frames and the disappeared detection frames in different working areas based on the association relationship between the multiple cameras to obtain association information; and updating the global target trajectory based on the association information.

[0072] Specifically, the association relationship between multiple cameras is, for example, that area X and area Y are adjacent at a specified position, so that when the target moves, it can cross different working areas from the adjacent position based on the above association relationship. The association information refers to the association between a newly added detection frame in a certain working area and a disappeared detection frame in another working area, that is, it corresponds to the same identity. In other words, for two working areas with adjacent boundaries, the disappearance order of multiple tracking targets can be first obtained at the adjacent boundary of one of the working areas, and the corresponding identification assignments and continuous tracking are performed for the multiple new targets that appear at the adjacent boundary in the other working area according to the above disappearance order.

[0073] For example, Figure 6 As shown, point b in area Y represents Figure 5 The ground coordinates of the detection frame b are shown in . If the local tracking information corresponding to the camera 201 indicates that there is no matching target for the detection frame point b, that is, there is a new target in the area Y; and the local tracking information corresponding to the camera 201 indicates that the continuously tracked "target 1" disappears in the current detection frame, that is, there is a disappeared target in the area X, then the detection frame b can be labeled "target 1" and the ground coordinates of point b can be added to the existing track of "target 1" (i.e. Figure 6 ), achieving target tracking across cameras and working areas.

[0074] Thus, according to the target tracking method based on multiple cameras in the embodiment of the present invention, by sequentially performing image detection on the current frames to be tested from each camera, and then performing global tracking in the monitoring area based on the detection results corresponding to each camera, the target object in the multi-channel monitoring video can be globally tracked based on fewer computing resources, thereby reducing the demand for computing resources. For example, it is not necessary to separately provide GPU computing resources for tracking target objects in each local area for each camera, but it is possible to provide fewer computing resources for global tracking of target objects in the monitoring area.

[0075] Based on the same technical concept, an embodiment of the present invention further provides a target tracking device for executing the target tracking method provided by any of the above embodiments. Figure 7 A schematic diagram of the structure of a target tracking device provided by an embodiment of the present invention.

[0076] like Figure 7 As shown, the apparatus 700 includes:

[0077] The acquisition unit 701 is used to acquire the current frames to be tested of multiple cameras arranged in the monitoring area;

[0078] A detection unit 702 is used to perform target detection on a current frame to be detected of each camera in turn, and obtain a detection frame set corresponding to each camera;

[0079] The tracking unit 703 is used to track the target according to the detection frame set corresponding to each camera, and determine the global target trajectory according to the tracking result.

[0080] In some possible implementations, the device 700 also includes: a frame selection unit, used to determine multiple frame numbers to be tested, and iteratively obtain current frames to be tested of multiple cameras in a time sequence according to the multiple frame numbers to be tested, thereby iteratively performing target tracking; wherein, an initial global target trajectory is obtained according to the initial frame number to be tested among the multiple frame numbers to be tested; and an iteratively updated global target trajectory is obtained according to the subsequent frame numbers to be tested among the multiple frame numbers to be tested.

[0081] In some possible implementations, the detection unit 702 is further used to: input the current frame to be detected of each camera into a target detection model for target detection; wherein the target detection model is a pedestrian detection model obtained through neural network training.

[0082] In some possible implementations, the detection unit 702 is further used to: after obtaining the detection frame set corresponding to each camera, perform a projection transformation on the bottom center point of each detection frame in the detection frame set corresponding to each camera according to the framing position of each camera, so as to determine the ground coordinates of each detection frame.

[0083] In some possible implementations, the viewing areas of multiple cameras at least partially overlap, and the device 700 is also used to: divide the working area of ​​each camera in the ground coordinate system according to the viewing area of ​​each camera; wherein the working areas of each camera do not overlap with each other, and if the ground coordinates of any detection frame corresponding to the first camera among the multiple cameras exceed the corresponding working area, then remove the any detection frame in the detection frame set of the first camera.

[0084] In some possible implementations, the detection unit 702 is further configured to: cut off a non-critical area in a working area of ​​each camera.

[0085] In some possible implementations, the tracking unit 703 is further used to: adopt a multi-target tracking algorithm, and perform multi-target tracking based on a detection frame set corresponding to each camera, and determine the local tracking information corresponding to each camera; wherein the parameters adopted for multi-target tracking are determined based on the historical test frames of each camera.

[0086] In some possible implementations, the multi-target tracking algorithm is a deepsort algorithm.

[0087] In some possible implementations, the tracking unit 703 is further used to: add an identity to each detection frame according to the local tracking information corresponding to each camera; and determine an iteratively updated global target trajectory based on the identity and ground coordinates of each detection frame.

[0088] In some possible implementations, the tracking unit 703 is further used to: determine the association relationship between multiple cameras based on the working areas of the multiple cameras; determine the newly added detection frame and the disappeared detection frame in the corresponding working area based on the local tracking information of each camera; associate the newly added detection frame and the disappeared detection frame in different working areas based on the association relationship between the multiple cameras to obtain association information; and determine the iteratively updated global target trajectory based on the association information.

[0089] In this way, according to the target tracking device based on multiple cameras in the embodiment of the present invention, by sequentially performing image detection on the current frames to be detected from each camera, and then performing global tracking in the monitoring area based on the detection results corresponding to each camera, the target object in the multi-channel monitoring video can be globally tracked based on fewer computing resources, thereby reducing the demand for computing resources. For example, it is not necessary to provide GPU computing resources for tracking target objects in each local area for each camera separately, but it is possible to provide fewer computing resources for global tracking of target objects in the monitoring area.

[0090] It should be noted that the device in the embodiment of the present application can implement the various processes of the embodiment of the aforementioned method and achieve the same effects and functions, which will not be repeated here.

[0091] Based on the same technical concept, an embodiment of the present invention also provides a target tracking system, which specifically includes: multiple cameras arranged in a monitoring area, and a target tracking device respectively communicatively connected to the multiple cameras; wherein the target tracking device is configured to execute the target tracking method provided in any of the above embodiments.

[0092] Based on the same technical concept, those skilled in the art can understand that various aspects of the present invention can be implemented as devices, methods or computer-readable storage media. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, which can be collectively referred to as "circuit", "module" or "device" here.

[0093] In some possible implementations, a target tracking device of the present invention may include at least one or more processors and at least one memory. The memory stores a program, and when the program is executed by the processor, the processor executes the following steps: Figure 1 The steps shown are: obtaining the current frames to be tested of multiple cameras set in the monitoring area; performing target detection on the current frames to be tested of each camera in the multiple cameras in turn to obtain a detection frame set corresponding to each camera; tracking the target according to the detection frame set corresponding to each camera, and determining the global target trajectory according to the tracking results.

[0094] Refer to the following Figure 8 The target tracking device 8 according to this embodiment of the present invention will be described. Figure 8 The device 8 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0095] like Figure 8 As shown, the apparatus 8 may be in the form of a general-purpose computing device, including but not limited to: at least one processor 10, at least one memory 20, and a bus 60 connecting different device components.

[0096] The bus 60 includes a data bus, an address bus, and a control bus.

[0097] The memory 20 may include a volatile memory, such as a random access memory (RAM) 21 and / or a cache memory 22 , and may further include a read only memory (ROM) 23 .

[0098] The memory 20 may also include a program module 24, such program module 24 includes but is not limited to: an operating device, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0099] The device 5 can also communicate with one or more external devices 2 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more other devices. Such communication can be performed through an input / output (I / O) interface 40 and displayed on a display unit 30. In addition, the device 5 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 50. As shown, the network adapter 50 communicates with other modules in the device 5 through a bus 60. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the device 5, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID devices, tape drives, and data backup storage devices.

[0100] Fig. 9 A computer-readable storage medium is shown for executing the method described above.

[0101] In some possible implementations, various aspects of the present invention may also be implemented in the form of a computer-readable storage medium, which includes program code. When the program code is executed by a processor, the program code is used to enable the processor to perform the method described above.

[0102] The method described above includes multiple operations and steps shown and not shown in the above figures, which will not be described in detail here.

[0103] The computer readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0104] like Fig. 9As shown, a computer-readable storage medium 90 according to an embodiment of the present invention is described, which can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the computer-readable storage medium of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution device, an apparatus, or a device.

[0105] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Python, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0106] In addition, although the operations of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0107] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A target tracking method, characterized in that: include: Obtain the current frames to be tested of multiple cameras set in the monitoring area; Performing target detection on the current frame to be detected of each camera in the multiple cameras in turn to obtain a detection frame set corresponding to each camera; The detection frame set includes a plurality of detection frames for representing position information and size information of the target; Performing target tracking according to the detection frame set corresponding to each camera, and determining a global target trajectory according to the tracking result; The global target trajectory is obtained by tracking subsequent frames to be tested of the camera using a multi-target detection algorithm and a target frame with an identified identity for each camera; After obtaining the detection frame set corresponding to each camera, and before performing target tracking according to the detection frame set corresponding to each camera, the method further includes: Performing a projection transformation on the center point of the bottom of each detection frame in the detection frame set corresponding to each camera according to the framing position of each camera, thereby determining the ground coordinates of each detection frame, so as to filter the detection frame set according to the ground coordinates; When the viewing areas of multiple cameras at least partially overlap, the detection frame set is filtered according to the following steps: Dividing the working area of ​​each camera in the ground coordinate system according to the framing area of ​​each camera; Among them, the working areas of each camera do not overlap with each other. If the ground coordinates of any detection frame corresponding to the first camera among the multiple cameras exceed the corresponding working area, the any detection frame is removed from the detection frame set of the first camera.

2. The method according to claim 1, characterized in that Also includes: Determine a plurality of sequence numbers of frames to be tested, and iteratively obtain the current frames to be tested of the plurality of cameras in a time sequence according to the plurality of sequence numbers of frames to be tested, thereby iteratively performing the target tracking; The initial global target trajectory is obtained according to the initial frame number to be tested among the multiple frame numbers to be tested; and the iteratively updated global target trajectory is obtained according to the subsequent frame numbers to be tested among the multiple frame numbers to be tested.

3. The method according to claim 2, characterized in that Performing target detection on the current frame to be tested of each camera, including: Inputting the current frame to be tested of each camera into the target detection model to perform the target detection; The target detection model is a pedestrian detection model obtained through neural network training.

4. The method according to claim 1, characterized in that: The method further comprises: The non-critical area in the working area of ​​each camera is cut off.

5. The method according to claim 2, characterized in that: Tracking is performed according to the detection frame set corresponding to each camera, including: Using a multi-target tracking algorithm, and performing multi-target tracking based on a set of detection frames corresponding to each camera, to determine local tracking information corresponding to each camera; The parameters used for the multi-target tracking are determined based on the historical frames to be tested of each camera.

6. The method according to claim 5, characterized in that The multi-target tracking algorithm is a deepsort algorithm.

7. The method according to claim 5, characterized in that Also includes: Adding an identity tag to each detection frame according to the local tracking information corresponding to each camera; The iteratively updated global target trajectory is determined based on the identity identifier and ground coordinates of each detection frame.

8. The method according to claim 5, characterized in that Also includes: Determine the association relationship between the multiple cameras according to the working areas of the multiple cameras; Determine a newly added detection frame and a disappeared detection frame in a corresponding working area according to the local tracking information of each camera; Associating the newly added detection frames and the disappeared detection frames in different working areas according to the association relationship between the multiple cameras to obtain association information; The global target trajectory after iterative update is determined according to the association information.

9. A target tracking device, characterized in that: include: An acquisition unit, used for acquiring current frames to be tested of multiple cameras arranged in a monitoring area; A detection unit, configured to perform target detection on a current frame to be detected of each camera in the plurality of cameras in turn, to obtain a detection frame set corresponding to each camera; The detection frame set includes a plurality of detection frames for representing position information and size information of the target; A tracking unit, configured to track a target according to a set of detection frames corresponding to each camera, and determine a global target trajectory according to the tracking result; The global target trajectory is obtained by tracking subsequent frames to be tested of the camera using a multi-target detection algorithm and a target frame with an identified identity for each camera; The detection unit is further used for: After obtaining the detection frame set corresponding to each camera, before performing target tracking according to the detection frame set corresponding to each camera, a projection transformation is performed on the center point of the bottom of each detection frame in the detection frame set corresponding to each camera according to the framing position of each camera, so as to determine the ground coordinates of each detection frame; When the framing areas of multiple cameras at least partially overlap, the detection unit is specifically configured to filter the detection frame set according to the following steps: Dividing the working area of ​​each camera in the ground coordinate system according to the framing area of ​​each camera; Among them, the working areas of each camera do not overlap with each other. If the ground coordinates of any detection frame corresponding to the first camera among the multiple cameras exceed the corresponding working area, the any detection frame is removed from the detection frame set of the first camera.

10. The device according to claim 9, characterized in that Also includes: A frame selection unit, used for determining a plurality of sequence numbers of frames to be tested, and iteratively acquiring the current frames to be tested of the plurality of cameras in a time sequence according to the plurality of sequence numbers of frames to be tested, thereby iteratively performing the target tracking; The initial global target trajectory is obtained according to the initial frame number to be tested among the multiple frame numbers to be tested; and the iteratively updated global target trajectory is obtained according to the subsequent frame numbers to be tested among the multiple frame numbers to be tested.

11. The device according to claim 10, characterized in that The detection unit is further used for: Inputting the current frame to be tested of each camera into the target detection model to perform the target detection; The target detection model is a pedestrian detection model obtained through neural network training.

12. The device according to claim 9, characterized in that The detection unit is further used for: The non-critical area in the working area of ​​each camera is cut off.

13. The device according to claim 10, characterized in that The tracking unit is further used for: Using a multi-target tracking algorithm, and performing multi-target tracking based on a set of detection frames corresponding to each camera, to determine local tracking information corresponding to each camera; The parameters used for the multi-target tracking are determined based on the historical frames to be tested of each camera.

14. The device according to claim 13, characterized in that The multi-target tracking algorithm is a deepsort algorithm.

15. The device according to claim 13, characterized in that The tracking unit is further used for: Adding an identity tag to each detection frame according to the local tracking information corresponding to each camera; The iteratively updated global target trajectory is determined based on the identity identifier and ground coordinates of each detection frame.

16. The device according to claim 13, characterized in that The tracking unit is further used for: Determine the association relationship between the multiple cameras according to the working areas of the multiple cameras; Determine a newly added detection frame and a disappeared detection frame in a corresponding working area according to the local tracking information of each camera; Associating the newly added detection frames and the disappeared detection frames in different working areas according to the association relationship between the multiple cameras to obtain association information; The global target trajectory after iterative update is determined according to the association information.

17. A target tracking system, characterized in that: include: A plurality of cameras disposed in the monitoring area, and a target tracking device respectively connected to the plurality of cameras in communication; Wherein, the target tracking device is configured to execute the method as described in any one of claims 1 to 8.

18. A target tracking device, characterized in that: include: One or more multi-core processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more multi-core processors, the one or more multi-core processors are enabled to implement: Obtain the current frames to be tested of multiple cameras set in the monitoring area; Performing target detection on the current frame to be detected of each camera in the multiple cameras in turn to obtain a detection frame set corresponding to each camera; The detection frame set includes a plurality of detection frames for representing position information and size information of the target; Performing target tracking according to the detection frame set corresponding to each camera, and determining a global target trajectory according to the tracking result; The global target trajectory is obtained by tracking subsequent frames to be tested of the camera using a multi-target detection algorithm and a target frame with an identified identity for each camera; After obtaining the detection frame set corresponding to each camera, and before performing target tracking according to the detection frame set corresponding to each camera, the method further includes: Performing a projection transformation on the center point of the bottom of each detection frame in the detection frame set corresponding to each camera according to the framing position of each camera, thereby determining the ground coordinates of each detection frame, so as to filter the detection frame set according to the ground coordinates; When the viewing areas of multiple cameras at least partially overlap, the detection frame set is filtered according to the following steps: Dividing the working area of ​​each camera in the ground coordinate system according to the framing area of ​​each camera; Among them, the working areas of each camera do not overlap with each other. If the ground coordinates of any detection frame corresponding to the first camera among the multiple cameras exceed the corresponding working area, the any detection frame is removed from the detection frame set of the first camera.

19. A computer-readable storage medium storing a program, wherein when the program is executed by a multi-core processor, the multi-core processor executes the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Cross-shot multi-target tracking method and system

    CN108876821A

  • Target detection tracking method and device, equipment and storage medium

    CN110428448A