Target tracking method and device, electronic equipment and readable storage medium

Abstract

The application is suitable for the technical field of monitoring, and provides a target tracking method and device, electronic equipment and readable storage medium, the method comprises: carrying out foreground identification on the acquired video frame to obtain the foreground area of each video frame; based on the foreground area of each video frame, generating the foreground track of the same foreground area; carrying out local matting detection on the foreground area in the foreground track to obtain the position detection result of the to-be-tracked target in the foreground area; based on the foreground track and the position detection result, tracking the to-be-tracked target in the video frame; based on the foreground track and the local matting, detecting the to-be-tracked target in the video frame, which can keep more stable tracking for the small target far away, reduce the tracking distance limit, and reduce the probability of missing the target far away.

Description

Technical Field

[0001] This application belongs to the field of surveillance technology, and in particular relates to a target tracking method, device, electronic device, and readable storage medium. Background Technology

[0002] With the development of surveillance technology, PTZ (pan-tilt-zoom) camera systems have become widely used. These systems typically consist of a panoramic bullet camera and a rotatable pan-tilt-zoom camera, offering the advantage of providing both panoramic views and coverage of localized details.

[0003] In some application scenarios, in order to cover a larger monitoring range, bullet camera images are often spliced ​​together using binocular or multi-view cameras. Moreover, the installation height of the equipment in such scenarios is relatively high, which makes distant targets occupy a smaller proportion in the image.

[0004] Currently, common camera-to-body (PTZ) tracking solutions primarily rely on panoramic target detection algorithms. These algorithms identify and locate the target within the panoramic view of the camera, then convert the camera's coordinates to obtain the corresponding PGN coordinates, and finally rotate the PGN to track the target. However, for distant and small targets, this approach is prone to problems such as unstable tracking, limited tracking distance, and missing distant targets. Summary of the Invention

[0005] This application provides a target tracking method, apparatus, electronic device, and readable storage medium, which can solve the problems of unstable tracking, limited tracking distance, and missing distant targets.

[0006] Firstly, this application provides a target tracking method, which may include:

[0007] Foreground detection is performed on the acquired video frames to obtain the foreground region of each video frame; based on the foreground region of each video frame, a foreground trajectory of the same foreground region is generated; local matting detection is performed on the foreground region in the foreground trajectory to obtain the position detection result of the target to be tracked in the foreground region; based on the foreground trajectory and the position detection result, the target to be tracked in the video frame is tracked.

[0008] In one possible implementation of the first aspect, before performing foreground detection on the acquired video frames to obtain the foreground region of each video frame, the method further includes:

[0009] Acquire a panoramic image of the target scene; perform size transformation on the panoramic image to obtain video frames.

[0010] In one possible implementation of the first aspect, foreground detection is performed on the acquired video frames to obtain the foreground region of each video frame, including:

[0011] The pixel brightness of the acquired video frame is compared with that of historical video frames to obtain the foreground area of ​​the video frame; where historical video frames are video frames acquired before the video frame.

[0012] In one possible implementation of the first aspect, a foreground trajectory of the same foreground region is generated based on the foreground region of each video frame, including:

[0013] Connecting the center points of the same foreground region in multiple consecutive video frames yields the foreground trajectory; the foreground trajectory includes trajectory identifier and foreground region information.

[0014] In one possible implementation of the first aspect, local matting detection is performed on the foreground region in the foreground trajectory to obtain the position detection result of the target to be tracked in the foreground region, including:

[0015] The foreground region is preprocessed, and then local matting is performed on the preprocessed foreground region to obtain a first local image containing the target to be tracked; target detection is performed on the first local image to obtain the position detection result of the target to be tracked.

[0016] In one possible implementation of the first aspect, after performing target detection on the first local image to obtain the position detection result of the target to be tracked, the method further includes:

[0017] The position detection result corresponding to the current video frame is compared with the position detection result corresponding to the historical video frames to obtain the change value of the detection box; if the change value of the detection box is within the preset threshold range, the target to be tracked is tracked based on the position detection result of the current video frame.

[0018] In one possible implementation of the first aspect, the method further includes:

[0019] After obtaining the first position detection result of the target to be tracked in the first video frame, trajectory prediction is performed based on the first position detection result to obtain the position prediction result of the target to be tracked at the next moment; based on the position prediction result, local matting processing is performed on the second video frame to obtain the second local image of the second video frame; target detection is performed on the second local image to obtain the second position detection result of the target to be tracked; based on the second position detection result, the target to be tracked in the second video frame is tracked; wherein, the first video frame and the second video frame are two adjacent video frames in the foreground trajectory.

[0020] Secondly, embodiments of this application provide a target tracking device, which may include:

[0021] The first detection unit is used to perform foreground detection on the acquired video frames to obtain the foreground region of each video frame;

[0022] A trajectory unit is used to generate a foreground trajectory for the same foreground region based on the foreground region of each video frame;

[0023] The second detection unit is used to perform local matting detection on the foreground region in the foreground trajectory to obtain the position detection result of the target to be tracked in the foreground region.

[0024] The tracking unit is used to track the target in the video frame based on the foreground trajectory and position detection results.

[0025] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0027] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the method described in the first aspect above.

[0028] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect, and will not be repeated here.

[0029] Compared with related technologies, the beneficial effects of this application are as follows: the embodiments of this application obtain the foreground region of each video frame by performing foreground recognition on the acquired video frames; based on the foreground region of each video frame, a foreground trajectory of the same foreground region is generated; local matting detection is performed on the foreground region in the foreground trajectory to obtain the position detection result of the target to be tracked in the foreground region; based on the foreground trajectory and the position detection result, the target to be tracked in the video frame is tracked; based on the foreground trajectory and the target detected by local matting, the target in the video frame can be detected, which can maintain more stable tracking for small targets at a distance, reduce the tracking distance limitation, and reduce the probability of missing distant targets; it has strong ease of use and practicality. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram illustrating an application scenario of the monitoring equipment provided in the embodiments of this application;

[0032] Figure 2 This is a flowchart illustrating the target tracking method provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of the foreground trajectory provided in the embodiments of this application;

[0034] Figure 4 This is a schematic diagram of image matting detection provided in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the change in the detection frame provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of trajectory prediction provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of the processing flow of each module provided in the embodiments of this application;

[0038] Figure 8 This is a schematic diagram of the target tracking device provided in the embodiments of this application;

[0039] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0040] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0041] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0042] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0043] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0044] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0045] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0046] Currently, in the surveillance field, surveillance equipment uses active tracking to identify and locate targets in panoramic images, and then tracks them. Especially for common camera-PTZ (pan-tilt-zoom) tracking scenarios, after acquiring a panoramic view, the bullet camera identifies and locates the target, then converts its coordinates to obtain the corresponding coordinates for the PTZ camera, and tracks the target by rotating the PTZ camera.

[0047] However, relying on traditional foreground detection algorithms to detect panoramic images results in poor recognition of small targets in the distance, leading to short trigger distances for tracking events and thus a short range for stable target tracking. While self-organizing background detection and background modeling methods or foreground detection methods can trigger long-range tracking, these methods treat all moving objects in the image as foreground and cannot perform fine-grained identification, thus failing to selectively track specific targets.

[0048] Therefore, in related technologies, target detection algorithms that rely on panoramic views are unstable in detecting small targets at a distance, which makes it easy to miss distant targets and causes the problem of limited tracking distance, thus limiting the application scenarios of gun-ball linkage; and traditional foreground detection algorithms for panoramic images cannot accurately distinguish the types of tracking targets, which can cause false triggering or false tracking.

[0049] To address the aforementioned problems, this application provides a target tracking method. The specific implementation process of this target tracking method is described below through embodiments.

[0050] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the monitoring equipment provided in the embodiments of this application. For example... Figure 1 As shown, the monitoring equipment can be a PTZ camera combined with a panoramic camera. This type of camera generally includes a panoramic camera and a rotatable PTZ camera. The panoramic camera acquires panoramic images and identifies and locates targets, while the rotatable PTZ camera performs rotation tracking. It has the advantage of monitoring both panoramic images and coverage of local details.

[0051] Linked tracking is a key function of PTZ cameras, and it can include manual tracking and event-driven tracking. Manual tracking involves selecting any position or area in the camera's panoramic view after system calibration. The camera will then rotate to the corresponding position and zoom in appropriately to reveal local details. Event-driven tracking, on the other hand, involves setting certain areas of the camera's monitoring screen as alert zones and enabling intelligent event detection after system calibration. When a target enters this zone and triggers the intelligent detection event, the camera will rotate to track the target and continue rotating in the target's direction of travel until the target disappears or exceeds the maximum tracking distance.

[0052] For example, bullet cameras have the characteristics of wide viewing angle and large monitoring range. In some special application scenarios, such as overpasses and open squares, in order to cover a larger monitoring range, bullet camera images often use binocular or multi-view splicing to obtain a larger monitoring range. Moreover, in such scenarios, the installation height of the equipment is also relatively high, so distant targets occupy a smaller proportion in the image.

[0053] It should be noted that, Figure 1 This is merely an illustrative description of the types of monitoring equipment and does not constitute a limitation on the types of monitoring equipment used. Figure 1 The monitoring equipment mentioned can also be other types of equipment.

[0054] Based on the above application scenarios, this application provides a target tracking method. The specific process of implementing this method is described below through embodiments of this application.

[0055] Please see Figure 2 , Figure 2 This is a flowchart illustrating a target tracking method provided in an embodiment of this application. It can be applied to the aforementioned monitoring equipment and can solve the problems of short tracking distance, unstable tracking, and easy omission of small targets in related technologies. Figure 2As shown, using the aforementioned monitoring equipment as the executing entity, the method includes the following steps:

[0056] S201, perform foreground detection on the acquired video frames to obtain the foreground region of each video frame.

[0057] In some embodiments, the monitoring device can capture continuous video frames through a camera and perform foreground detection on the continuous video frames to obtain the foreground region of each video frame.

[0058] For example, multiple video frames can be captured by a camera in a monitoring device, and the foreground region of each video frame can be obtained through a foreground detection algorithm. The information of the foreground region may include the timestamp of the video frame and the location information of the foreground region; for example, the location information of the foreground region can be represented by coordinates (x, y, w, h), where x and y represent the coordinates of a vertex of the foreground region, and w and h represent the distances of the foreground region in the coordinate system based on the aforementioned vertex coordinates, corresponding to the x-axis and y-axis directions, respectively.

[0059] In some embodiments, before performing foreground detection on the acquired video frames to obtain the foreground region of each video frame, the method further includes:

[0060] Acquire a panoramic image of the target scene; perform size transformation on the panoramic image to obtain video frames.

[0061] For example, in order to reduce the computational load, the size (pixels or dimensions) of the video frame used for foreground detection can be smaller than the size of the original image captured by the machine gun. Therefore, after obtaining a series of panoramic images of the target scene through the machine gun, the size of each panoramic image is transformed (e.g., image cropping) to obtain a video frame suitable for foreground detection.

[0062] In some embodiments, foreground detection is performed on the acquired video frames to obtain the foreground region of each video frame, including:

[0063] The pixel brightness of the acquired video frame is compared with that of historical video frames to obtain the foreground region of the video frame; where historical video frames are video frames acquired before the video frame for which foreground detection is performed.

[0064] For example, a background modeling algorithm is used to detect the foreground region of each video frame. This can be achieved by comparing the brightness values ​​of each pixel in the current video frame with previously acquired historical video frames to obtain the brightness difference for each pixel. Based on this brightness difference and a preset brightness difference threshold, it is determined whether to include the compared pixels in the current video frame as pixels in the foreground region. The historical video frame can be the previous frame or multiple adjacent video frames.

[0065] Using the above method, all foreground regions in the current video frame can be detected, such as... Figure 3 The images show the foreground regions in video frames 1 through 5; it should be noted that... Figure 3 For illustrative purposes only, the foreground region detected in each video frame may include multiple regions. For example, the area where a moving object is located in each frame may be identified as the foreground region, thereby determining the foreground region (or motion region) in the current video frame based on historical video frames. The detection of the foreground region of a video frame has the characteristics of long detection distance and high sensitivity.

[0066] S202, based on the foreground region of each video frame, generates a foreground trajectory for the same foreground region.

[0067] In some embodiments, the monitoring device (e.g., a bullet camera) can maintain the foreground region in consecutive video frames and generate foreground trajectories. Each video frame may include multiple foreground regions, and for each of the multiple foreground regions, foreground trajectories for the same foreground region in consecutive video frames can be generated, that is, multiple foreground trajectories corresponding to multiple foreground regions can be generated.

[0068] For example, such as Figure 3 As shown, in video frames 1 to 5, foreground trajectories are generated for the same foreground region; among them, Figure 3 As an example only, each video frame may also include multiple foreground regions, thereby generating multiple foreground trajectories for consecutive video frames.

[0069] In some embodiments, a foreground trajectory for the same foreground region is generated based on the foreground region of each video frame, including:

[0070] Connecting the center points of the same foreground region in multiple consecutive video frames yields the foreground trajectory; the foreground trajectory includes trajectory identifier and foreground region information.

[0071] For example, the forward trajectory is a queue of the same foreground region in consecutive video frames. The position or size of the foreground region may change in consecutive video frames as objects in the frame move. Therefore, the foreground trajectory is obtained by connecting the center points of the same foreground region in each video frame.

[0072] In this context, the same foreground region can be defined as an area composed of pixels with the same brightness value in the foreground region of the image. For moving objects, the position of the area corresponding to pixels with the same brightness value changes as the acquisition time changes. Therefore, each foreground trajectory can correspond to a unique trajectory identifier and foreground region information.

[0073] For example, the trajectory identifier may include a trajectory identity identifier and a trajectory generation timestamp, such as foreground trajectory 1 and the specific generation time of foreground trajectory 1. Foreground region information can be represented by a rectangle, and the foreground region in each video frame has its corresponding area, which can be identified by the left, top, right, and bottom coordinates of the rectangle.

[0074] By forming a foreground trajectory based on the foreground region of continuous video frames, it is convenient to perform a series of maintenance tracking on the area where the target to be tracked is located; when there are multiple targets to be tracked, the foreground trajectory corresponding to each different target can be identified and tracked, reducing the probability of mistracking.

[0075] S203, perform local matting detection on the foreground region in the foreground trajectory to obtain the position detection result of the target to be tracked in the foreground region.

[0076] In some embodiments, local matting detection is performed on the foreground region corresponding to the video frame in the foreground trajectory to identify the position detection result of the target to be tracked in the video frame.

[0077] For example, the monitoring device (camera) can parse the foreground trajectory (foreground region queue), expand or scale the foreground region box, and adjust the size of the foreground region box to match the size of the input image of the intelligent detection algorithm; such as Figure 4 As shown, when the foreground region box is smaller than the size of the input image of the intelligent detection algorithm (the foreground region box is small), the foreground region box can be enlarged or expanded. When the foreground region box is larger than the size of the input image of the intelligent detection algorithm (the foreground region box is large), the foreground region box can be shrunk.

[0078] In some embodiments, local matting detection is performed on the foreground region in the foreground trajectory to obtain the position detection result of the target to be tracked in the foreground region, including:

[0079] The foreground region is preprocessed, and then local matting is performed on the preprocessed foreground region to obtain a first local image containing the target to be tracked; target detection is performed on the first local image to obtain the position detection result of the target to be tracked.

[0080] For example, the monitoring device can traverse and perform local matting detection on the foreground region queue of each foreground trajectory. The foreground region is preprocessed to filter out abnormal foreground region boxes, and the foreground trajectories are initially screened, with the size of the foreground region for each foreground trajectory being filtered; for example, foreground regions smaller than a foreground region threshold are removed, and the corresponding foreground trajectories are considered invalid and will not be traversed or subjected to local matting detection; for valid foreground trajectories, the foreground region corresponding to the foreground region (the foreground region box meets the foreground region threshold) will be scaled or expanded, such as... Figure 4 As shown, the region containing the target (or foreground target) to be tracked is locally matted to obtain a first local image. Then, the image is scaled to complete the local matting detection, resulting in the position detection result of the target to be tracked. Accordingly, after the camera detects the position detection result of the target to be tracked, it can send it to the rotatable pan-tilt camera for real-time tracking of the target.

[0081] For example, before determining the location detection result of the target to be tracked, the monitoring device performs forward inference detection based on the input first local image, identifies targets such as human figures / vehicles that may exist in the foreground area that need to be tracked, and records the target detection result corresponding to the foreground trajectory; in the subsequent identification process, the device predicts and calculates information such as the direction of travel of the target detection result based on the foreground trajectory, so as to facilitate continuous and stable tracking of the target.

[0082] For example, forward inference-based detection detects whether there are target types to be tracked in the foreground region, such as people, vehicles, and non-motorized vehicles. A lightweight target detection network is used to improve the original YOLOv4-tiny model architecture by adding a CSP (Cross Stage Partial) structure between the backbone and the detection head, deepening the original network structure and improving the model's learning and feature extraction capabilities. The target detection results output by forward inference-based detection can include location information (such as a rectangular detection box representing the location detection result), the target detection category, and the confidence score.

[0083] Meanwhile, when multiple targets exist, stable tracking of the same detected target is required. However, when the foreground region contains more than one target (such as a crowded group of people, or two people walking towards each other whose trajectories intersect), target detection is needed to distinguish them. When detecting the position of a target, the position information of the detection box of the tracked target in the previous frame is retained. The detection result of the current frame is compared with the detection box of the previous frame one by one to calculate the Intersection over Union (IoU). The detection box with the largest IoU and the same target type is selected. Detection boxes with an IoU of 0 or different target types are filtered out to achieve single target (same target) detection and reduce the probability of trajectory misconnection during tracking.

[0084] In some embodiments, after performing target detection on the first local image to obtain the location detection result of the target to be tracked, the method further includes:

[0085] The position detection result corresponding to the current video frame is compared with the position detection result corresponding to the historical video frames to obtain the change value of the detection box; if the change value of the detection box is within the preset threshold range, the target to be tracked is tracked based on the position detection result of the current video frame.

[0086] For example, to reduce the probability of misaligned tracks or mistracked targets, the monitoring device can filter the position detection results corresponding to the current video frame. The position detection results can be represented by the size and position of the output detection box. The size of the detection box corresponds to the distance of the target to be tracked. For different video frames, the same target to be tracked will have different detection box sizes at different positions or distances. Therefore, the size of the detection box of the target to be tracked in the current video frame can be compared with the size of the detection boxes in historical video frames. If the size change exceeds a preset threshold, the target to be tracked in the current video frame will not be tracked. For example, the detection box of the current video frame can be compared with the detection boxes of the previous five video frames. If the size change of the detection box of the current video frame exceeds 30% of the average size of the detection boxes in the previous five frames, the target to be tracked in the current video frame will not be tracked. If the change is less than 30%, tracking will continue based on the position detection results of the current video frame; thus reducing the probability of mistracking the target.

[0087] like Figure 5 As shown, when detection box 1 changes to detection box 2, based on the judgment of the change in the size of the detection box, if it is determined that detection box 2 meets the preset threshold, then the target will continue to be tracked based on the position detection result corresponding to detection box 2; when detection box 1 changes to detection box 3, if the change in the size of the detection box exceeds the preset threshold, then the target tracking will no longer continue based on the position detection result corresponding to detection box 3.

[0088] It should be noted that, Figure 5 The changes in the detection bounding box are merely illustrative examples; it is understood that different detection bounding boxes correspond to different video frames. Figure 5 The video frames shown are for illustrative purposes only. In cases of misconnected trajectories or mistracked targets, judgments can also be made based on changes in the position of the detection box. Since the position change of the detection box is compared with the position change predicted based on the trajectory, if it exceeds the threshold range, target tracking can be performed without using the current detection box.

[0089] Since the size of the target being tracked may vary, meaning the size of the detection box changes in real time according to the size of the target, if the target moves rapidly into the distance, the size of the detection box may change significantly in adjacent frames. By using the moving average of the detection box, the probability of tracking anomalies is reduced due to the current target being mistakenly jumped to a distant target when the trajectories of two targets intersect.

[0090] S204 tracks the target in the video frame based on the foreground trajectory and position detection results.

[0091] For example, target tracking is a continuous process that requires stable tracking of the target until it disappears. By detecting the position of the target to be tracked in the current video frame and predicting the target's direction of travel based on the foreground trajectory, the possible positions of the target to be tracked in the next video frame are detected again, and the position detection results of the target to be tracked are updated in real time, so as to achieve stable and continuous tracking of the target.

[0092] In one possible implementation, the method further includes: after obtaining the first position detection result of the target to be tracked in the first video frame, performing trajectory prediction based on the first position detection result to obtain the position prediction result of the target to be tracked at the next moment; performing local matting processing on the second video frame based on the position prediction result to obtain a second local image of the second video frame; performing target detection on the second local image to obtain a second position detection result of the target to be tracked; and tracking the target to be tracked in the second video frame based on the second position detection result; wherein the first video frame and the second video frame are two adjacent video frames in the foreground trajectory.

[0093] For example, such as Figure 6As shown, the first video frame and the second video frame can be two consecutive video frames. After detecting the target to be tracked in the first video frame and obtaining the target detection result, based on the first position detection result and the foreground trajectory in the target detection result, the possible position of the target to be tracked in the next moment (the next video frame) is predicted to obtain the position prediction result. Based on the position prediction result, local matting is performed on the second video frame to obtain the second local image. Then, the matted image is used for detection to obtain the target detection result corresponding to the second video frame, such as the second position detection result of the target to be tracked in the second video frame. Based on the second position detection result, the target to be tracked in the second video frame is tracked, thus achieving continuous and stable tracking of the target to be tracked.

[0094] In some embodiments, during the movement of the target to be tracked, the camera of the monitoring device will synchronously send the position detection results to the PTZ camera. The PTZ camera will obtain the updated coordinates of the target to be tracked, and after coordinate system transformation, it will rotate to the target position for tracking.

[0095] For example, coordinate transformation is performed because the bullet camera and PTZ camera form a combined system, consisting of a fixed panoramic bullet camera and a movable PTZ camera (short for Pan / Tilt / Zoom, representing omnidirectional pan / tilt movement and lens zoom / zoom control). When a moving target is tracked in the bullet camera, the PTZ camera needs to rotate to a certain position so that the tracked target is imaged in the center of the PTZ camera's screen and maintains a certain size. Accordingly, the coordinates of any pixel in the bullet camera are mapped to the PTZ camera's screen (a mapping value is required). Based on this mapping value, the coordinates of any pixel in the bullet camera are transformed into the PTZ value of the target image at the center of the PTZ camera's screen. The calibration process of the bullet and PTZ camera is the process of establishing a correspondence between the bullet camera and the PTZ camera's screen. Generally, several pairs of calibration points can be selected for calibration. After calibration, a coordinate transformation matrix is ​​obtained, which is used to map the pixel positions in the bullet camera to the pixel positions in the PTZ camera's screen.

[0096] like Figure 7As shown in the diagram, this application embodiment provides a schematic diagram of the processing flow of each module. In the target tracking process implemented by each module, the foreground detection module detects video frames to obtain foreground region information; the foreground region information is input to the foreground trajectory module to generate a foreground trajectory queue; the foreground trajectory queue is input to the tracking module, which performs local matting based on the foreground trajectory to obtain a local image, and inputs the local image to the target detection module; the target detection module detects the target to be tracked to obtain a target detection result (including the type and location information of the target to be tracked); the target detection result is input to the trajectory prediction module, which predicts the trajectory of the target to be tracked based on the direction of travel of the target to be tracked; the trajectory of the target to be tracked is input to the tracking module for target tracking. The tracking module can also perform local matting on video frames based on the trajectory prediction result to obtain a local image, and input this local image to the target detection module for detection.

[0097] The embodiments of this application achieve high sensitivity in triggering tracking events. Foreground trajectories are characterized by short generation time and high trigger sensitivity. For small targets, the foreground detection algorithm outperforms the full-image intelligent detection algorithm, resulting in more sensitive target detection and a longer trigger distance. Simultaneously, the linkage tracking distance is extended. A gun-ball linkage scheme based on local matting is adopted, using a local region of the panoramic image as the detection object. With a fixed input to the detection model, using a local region as input results in a smaller image size transformation compared to a full-image input, ensuring the proportion of small targets in the image after scale transformation. This effectively detects small targets at a distance, thus achieving tracking at greater distances. Furthermore, the tracking process is more stable with fewer trajectory misconnections. The abnormal box filtering and single-target detection strategies employed in the gun-ball linkage tracking process using the local detection scheme prevent misconnections to other targets during tracking, ensuring the stability of the tracking process, making it more stable than other gun-ball linkage strategies.

[0098] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0099] Based on the above target tracking method, this application also provides a target tracking device, such as... Figure 8 As shown, the device may include:

[0100] The first detection unit 81 is used to perform foreground detection on the acquired video frames to obtain the foreground region of each video frame;

[0101] The trajectory unit 82 is used to generate a foreground trajectory for the same foreground region based on the foreground region of each video frame;

[0102] The second detection unit 83 is used to perform local matting detection on the foreground region in the foreground trajectory to obtain the position detection result of the target to be tracked in the foreground region.

[0103] The tracking unit 84 is used to track the target to be tracked in the video frame based on the foreground trajectory and the position detection result.

[0104] In one possible implementation, the device further includes an image processing unit for acquiring a panoramic image of the target scene and performing a size transformation on the panoramic image to obtain a video frame.

[0105] In one possible implementation, the first detection unit is further configured to compare the pixel brightness of the acquired video frame with that of a historical video frame to obtain the foreground region of the video frame; wherein the historical video frame is a video frame acquired before the video frame.

[0106] In one possible implementation, the trajectory unit is also used to connect the center points of the same foreground region in multiple consecutive video frames to obtain a foreground trajectory; wherein the foreground trajectory includes a trajectory identifier and foreground region information.

[0107] In one possible implementation, the second detection unit is further configured to preprocess the foreground region, perform local matting on the preprocessed foreground region to obtain a first local image containing the target to be tracked, and perform target detection on the first local image to obtain the position detection result of the target to be tracked.

[0108] In one possible implementation, the second detection unit is further configured to compare the position detection result corresponding to the current video frame with the position detection result corresponding to the historical video frames to obtain the detection box change value; if the detection box change value is within a preset threshold range, then the target to be tracked is tracked based on the position detection result of the current video frame.

[0109] In one possible implementation, the device further includes a trajectory prediction unit, which, after acquiring the first position detection result of the target to be tracked in the first video frame, performs trajectory prediction based on the first position detection result to obtain the position prediction result of the target to be tracked at the next moment.

[0110] In one possible implementation, the second detection unit is further configured to perform local matting processing on the second video frame based on the position prediction result to obtain a second local image of the second video frame; and to perform target detection on the second local image to obtain a second position detection result of the target to be tracked.

[0111] In one possible implementation, the tracking unit is further configured to track the target to be tracked in the second video frame based on the second position detection result; wherein the first video frame and the second video frame are two adjacent video frames in the foreground trajectory.

[0112] Through the embodiments of this application, foreground recognition is performed on the acquired video frames to obtain the foreground region of each video frame; based on the foreground region of each video frame, a foreground trajectory of the same foreground region is generated; local matting detection is performed on the foreground region in the foreground trajectory to obtain the position detection result of the target to be tracked in the foreground region; based on the foreground trajectory and the position detection result, the target to be tracked in the video frame is tracked; based on the foreground trajectory and the target detected by local matting, the target in the video frame can be detected, which can maintain more stable tracking for small targets at a distance, reduce the tracking distance limitation, and reduce the probability of missing distant targets.

[0113] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0114] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0115] Figure 9 This is a schematic diagram of the structure of an electronic device 9 provided in an embodiment of this application. Figure 9 As shown, the electronic device 9 of this embodiment includes: at least one processor 90 ( Figure 9 Only one is shown in the diagram), memory 91, and computer program 92 stored in memory 91 and executable on at least one processor 90, wherein the processor 90 executes computer program 92 to implement the steps in the above embodiments.

[0116] The electronic device 9 is a computing device capable of monitoring and tracking. The electronic device 9 may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 9 and does not constitute a limitation on electronic device 9. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0117] The processor 90 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0118] In some embodiments, the memory 91 may be an internal storage unit of the electronic device 9, such as a hard disk or memory of the electronic device 9. In other embodiments, the memory 91 may be an external storage device of the electronic device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 9. Furthermore, the memory 91 may include both internal and external storage units of the electronic device 9. The memory 91 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0120] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0122] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A target tracking method characterized by, The method comprises: foreground detection is performed on the obtained video frames to obtain a foreground region of each video frame; a foreground track of the same foreground region is generated based on the foreground region of each video frame; local matting detection is performed on the foreground region in the foreground track to obtain a position detection result of a target to be tracked in the foreground region; tracking is performed on the target to be tracked in the video frame based on the foreground track and the position detection result; the local matting detection on the foreground region in the foreground track to obtain the position detection result of the target to be tracked comprises: the foreground region is preprocessed, and the preprocessed foreground region is subjected to local matting processing to obtain a first local image containing the target to be tracked; target detection is performed on the first local image to obtain the position detection result of the target to be tracked; the method further comprises: after obtaining a first position detection result of the target to be tracked in a first video frame, trajectory prediction is performed based on the first position detection result to obtain a position prediction result of the target to be tracked at a next time; based on the position prediction result, local matting processing is performed on a second video frame to obtain a second local image of the second video frame; target detection is performed on the second local image to obtain a second position detection result of the target to be tracked; tracking is performed on the target to be tracked in the second video frame based on the second position detection result; wherein the first video frame and the second video frame are two adjacent video frames in the foreground track.

2. The method of claim 1, wherein, Before the foreground detection is performed on the obtained video frames to obtain the foreground region of each video frame, the method further comprises: obtaining a panoramic image of a target scene; performing size transformation processing on the panoramic image to obtain the video frames.

3. The method of claim 1, wherein, The foreground detection performed on the obtained video frames to obtain the foreground region of each video frame comprises: comparing the pixel brightness of the obtained video frame with that of a historical video frame to obtain the foreground region of the video frame; wherein the historical video frame is a video frame obtained before the video frame.

4. The method of claim 1, wherein, The foreground track of the same foreground region is generated based on the foreground region of each video frame, comprising: connecting the center points of the same foreground region in a plurality of continuous video frames to obtain the foreground track; wherein the foreground track comprises a track identifier and foreground region information.

5. The method of claim 1, wherein, After the target detection is performed on the first local image to obtain the position detection result of the target to be tracked, the method further comprises: comparing the position detection result corresponding to a current video frame with the position detection result corresponding to a historical video frame to obtain a detection box change value; if the detection box change value is within a preset threshold range, tracking is performed on the target to be tracked based on the position detection result of the current video frame.

6. A target tracking device, characterized by, The device comprises: a first detection unit configured to perform foreground detection on obtained video frames to obtain a foreground region of each video frame; a trajectory unit, configured to generate a foreground trajectory of the same foreground region based on the foreground region of each video frame; a second detection unit, configured to perform local matting detection on the foreground region in the foreground trajectory to obtain a position detection result of a target to be tracked in the foreground region; a tracking unit, configured to track the target to be tracked in the video frame based on the foreground trajectory and the position detection result; the second detection unit is further configured to perform preprocessing on the foreground region, perform local matting processing on the preprocessed foreground region to obtain a first local image containing the target to be tracked, and perform target detection on the first local image to obtain the position detection result of the target to be tracked; the device further includes a trajectory prediction unit, configured to perform trajectory prediction based on the first position detection result to obtain a position prediction result of the target to be tracked at a next time after obtaining the first position detection result of the target to be tracked in the first video frame; the second detection unit is further configured to perform local matting processing on a second video frame based on the position prediction result to obtain a second local image of the second video frame, perform target detection on the second local image to obtain a second position detection result of the target to be tracked, and the tracking unit is further configured to track the target to be tracked in the second video frame based on the second position detection result; wherein the first video frame and the second video frame are two adjacent video frames in the foreground trajectory. 7.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images