PTZ target tracking method suitable for digital control tower optical system

Abstract

The invention relates to a PTZ target tracking method suitable for a digital control tower optical system. The PTZ target tracking method comprises the following steps: accessing a dome camera video, executing single target tracking according to the initial position of a target, obtaining the position and size (x, y, w, h) of the target in each frame of image in real time, obtaining a target category k through a target detection technology, and according to the target information (x, y, w, h; the PTZ control of the dome camera is implemented according to the target information (x, y, w, h; and k) estimating the target motion angular velocity, and determining the rotating speed grade of the dome camera based on the estimated target motion angular velocity. According to the method, the target motion angular velocity is estimated by using image information and other prior knowledge, and the rotation speed of the dome camera is set according to the estimated value of the target angular velocity, so that the rotation of the dome camera can better adapt to target motion at different speeds, a relatively stable and continuous tracking effect can be obtained, and a control mode is effectively simplified.

Description

Technical Field

[0001] The invention relates to a PTZ target tracking method suitable for a digital tower optical system, and also relates to a PTZ target tracking system suitable for a digital tower optical system for implementing the method, belonging to the technical field of target tracking and PTZ control. Background Art

[0002] Digital towers are facilities that provide air traffic services to airports by replacing on-site visual observation with remote monitoring information. They can realize a unified, centralized, and standardized air traffic control support model, which helps to solve the problem of insufficient air traffic control support capabilities at small and medium-sized airports. With the increasing number of small and medium-sized airports in my country, the demand for the application of digital tower technology is increasing.

[0003] PTZ target tracking is a basic function required by the optical system of a digital tower. It uses visual tracking technology to locate the target of interest in the video image, and automatically adjusts the direction and focal length of PTZ cameras such as dome cameras according to the target position and size, so that the target is located in the center of the image field of view and sufficient details can be observed to facilitate decision-making and command by airport managers.

[0004] For digital tower applications, one of the main ways to implement PTZ tracking is to perform target tracking in the ball camera video, and control the ball camera's steering, rotation speed and focal length according to the target position and size. In the ball camera screen, by increasing the focal length, the target can be kept large enough for tracking, which is conducive to achieving smooth and continuous tracking effects. The key to this PTZ tracking is the ball camera speed control strategy. Currently, the speed of the ball camera is usually adjusted according to the position deviation of the target in the image. The closer the target is to the edge of the image, the faster the ball camera rotates. Since this control strategy only considers the target position and not the target speed, it may cause the ball camera speed to not match the target speed. Especially when the target is small and the ball camera speed is too fast, it is easy to lose the tracking target. Summary of the invention

[0005] The purpose of the present invention is to solve the above speed mismatch problem more effectively.

[0006] The technical solution of the present invention is: a PTZ target tracking method suitable for a digital tower optical system, which is connected to (based on) a ball camera video, performs single target tracking according to the initial position of the target, and obtains the position and size (x, y, w, h) of the target in each frame image in real time, wherein x and y are the coordinates of the center point of the target frame, w and h are the width and height of the target frame, respectively, obtains the target category k through target detection technology, implements PTZ control of the ball camera according to target information (x, y, w, h; k), and in the PTZ control of the ball camera, estimates the target motion angular velocity according to the target information (x, y, w, h; k), and determines the rotation speed level of the ball camera based on the estimated target motion angular velocity.

[0007] Preferably, the method of determining the rotation speed level of the ball camera based on the estimated target motion angular velocity is: based on the ball camera rotation speed function obtained by offline calibration, the rotation speed level variable l of the ball camera under the current target motion angular velocity is calculated, and the rotation speed level of the ball camera is set to the upper integer of l. Or round down The speed function of the ball machine is:

[0008] ω(l)=al 2 +bl+c,

[0009] Where ω(l) is the speed of the ball camera, which is taken as the angular velocity of the target motion, and a, b, and c are the corresponding fitting polynomial coefficients obtained by offline calibration.

[0010] Preferably, the method of estimating the angular velocity of the target movement is: presetting a reference state of the target category, measuring the size of the target in the image under the reference state, and calculating the real-time distance from the target to the ball camera (for example, the rotation center of the ball camera) based on the following formula:

[0011]

[0012] The target motion angular velocity is calculated based on the following formula:

[0013]

[0014] In the formula

[0015]

[0016]

[0017] d0, S0 and z0 are the distance from the target to the ball camera, the size of the target in the image and the zoom multiple of the ball camera in the reference state, respectively. d, S and z are the distance from the target to the ball camera, the size of the target in the image and the zoom multiple of the ball camera in the current state, respectively. w img and h img are the width and height of the image (can be counted in pixels, the same below), w0 and h0 are the width and height of the target in the image under the reference state, w and h are the width and height of the target in the image under the current state, av is the angular velocity of the target, v k Preset value for the movement speed of the corresponding target category (the target category corresponding to the target).

[0018] Based on the above, the size of the target in the image is the square root of the quotient of the target (target box) area divided by the image area, which is dimensionless.

[0019] In calculating the real-time distance from the target to the ball camera, the following formula can be used to calculate No need to calculate S0 and S separately:

[0020]

[0021] Preferably, the image is divided into nine regions vertically and horizontally, and the rotation direction (referred to as steering) of the ball camera and / or the zoom multiple of the ball camera are determined based on the region where the target is located.

[0022] Preferably, the upper and lower thresholds w of the central area in the image width direction of the nine areas are mi and w max Set to 0.4 and 0.6 times the image width w respectively img , the upper and lower thresholds h in the image height direction min and h max Set to 0.4 and 0.6 times the image height h respectively img .

[0023] The method of determining the rotation direction of the ball camera based on the area where the target is located is: when the target is in any of the eight areas of the image, namely, upper, lower, left, right, upper left, upper right, lower left and lower right, the direction of the ball camera is determined according to the direction of the area where the target is located relative to the central area (or in other words, the opposite direction), so that the target in the image moves from the area where it is located toward the central area. When the target is in the central area of ​​the image, the ball camera is not rotated.

[0024] Preferably, the method of determining the zoom multiple of the ball camera based on the area where the target is located is: when the target is in the central area of ​​the image, the focal length of the ball camera is determined or adjusted according to the target size, so that the target size in the image is maintained within or adjusted to a set target size range.

[0025] Furthermore, in the case where the target is in an area outside the central area of ​​the image, the focal length (zoom ratio) of the ball camera can be determined or adjusted according to the target size so that the target size in the image is maintained within or adjusted to a set target size range; the size of the target may also not be adjusted, that is, the focal length (zoom ratio) of the ball camera may be kept unchanged to simplify the adjustment process.

[0026] Preferably, the upper and lower thresholds S of the target size range are set min and S max They are 0.04 and 0.08 respectively.

[0027] Furthermore, if the target size S is smaller than S min , then increase the zoom factor Z of the ball camera; if the size S of the target is greater than S max , then reduce the zoom factor Z of the ball camera.

[0028] Preferably, the zoom factor Z increases or decreases step by step, each time by 0.5.

[0029] A PTZ target tracking system suitable for a digital tower optical system, which uses any PTZ target tracking method suitable for a digital tower optical system disclosed in the present invention to implement PTZ control of a ball camera, comprising:

[0030] The ball camera speed measurement calibration module is used to establish the functional relationship between the speed level and angular velocity of the ball camera (speed function);

[0031] The target detection and tracking module is used to track the specified target in the ball camera video, identify the target category, and obtain the target position and size in real time;

[0032] The ball camera PTZ control module is used to estimate the target movement angular velocity according to the target type, position and size, and use the rotation speed function obtained by offline calibration to implement PTZ control of the ball camera (including the so-called ball camera and other similar PTZ camera equipment) to set the rotation speed, steering and focal length of the ball camera.

[0033] Since the present invention uses image information and other prior knowledge to estimate the angular velocity of the target movement, and sets the rotation speed of the ball camera according to the estimated value of the target angular velocity, the rotation of the ball camera can better adapt to the movement of targets at different speeds, which is conducive to obtaining a relatively stable and continuous tracking effect; since the image is divided into nine areas aligned vertically and horizontally, corresponding steering and zoom controls are used according to the areas where the target is located, which effectively simplifies the control method and ensures the control effect.

[0034] In addition to digital towers, the present invention is also applicable to other video surveillance applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a process flow chart of the present invention;

[0036] Figure 2 is an example of an image region division diagram according to the present invention;

[0037] Figure 3 It is a comparison diagram between the fitted speed function curve and the measured speed curve involved in the present invention. DETAILED DESCRIPTION

[0038] The PTZ target tracking system of the present invention includes the following modules, which are used to implement the PTZ target tracking method of the present invention:

[0039] 1. Speed ​​measurement calibration module of ball camera. This module is an offline processing module, which is used to establish the functional relationship between the speed level and angular velocity of the ball camera;

[0040] 2. Target detection and tracking module. This module is used to track the specified target in the ball camera video, identify the target category, and obtain the target position and size in real time;

[0041] 3. PTZ control module for ball cameras. This module is used to estimate the target angular velocity according to the target type, position and size, and use the rotation speed function obtained by offline calibration to implement PTZ control of PTZ camera equipment (for example, ball cameras), and set the rotation speed, steering and focal length of the ball camera (or other camera equipment, the same below).

[0042] See also Figure 1 The main working process / processing flow of the present invention includes:

[0043] 1. Speed ​​measurement calibration of ball camera

[0044] A ball camera speed measurement calibration module is set up to establish a functional relationship between the ball camera rotation speed level (or speed level) and the angular velocity.

[0045] For a given ball camera, assume that it has L rotation speed levels (for example, L=7 for Hikvision ball camera), and the rotation angular velocity at different levels l is ω(l) (l=1, 2, ..., L).

[0046] The speed measurement calibration module of the ball camera uses the least square method to fit the speed function ω(l) based on the measured speed data at each speed level. Specifically, it includes:

[0047] 1) Determine the angular velocity of the ball camera at different levels: set the rotation speed level l and the rotation time t (for example, t is set to 5 seconds), and use the ball camera API to obtain its rotation angle α during this time. The angular velocity of the ball camera at level l is ω(l) = α / t. Usually, the horizontal rotation speed and vertical rotation speed of the ball camera are different, and the horizontal angular velocity ω can be measured separately. h (l) and the vertical angular velocity ω v (l), and take ω(l) as the mean of the two.

[0048] 2) Fitting the speed function: The angular velocity ω(l) (l=1, 2, ..., L) of the ball camera at different speed levels is known, and it is assumed that ω(l) satisfies a quadratic function.

[0049] ω(l)=al 2 +bl+c Formula 1 uses the least square method to fit ω(l) to obtain the function coefficients a, b, and c. Since l in the above quadratic function is a continuous variable, it can be called a speed level variable.

[0050] Figure 2 The fitting results of the speed function of a ball camera are given. The ball camera has 7 speed levels, where the solid line corresponds to the measured speed and the dotted line is the fitting function curve.

[0051] 2. Target Detection and Tracking

[0052] Set up the target detection and tracking module to track the specified target in the ball camera video, identify the target category, and obtain the target position and size in real time. Specifically include:

[0053] Access the ball camera video, perform single target tracking based on the target's initial position (which can be obtained through target detection or user selection), and obtain the target's position and size (x, y, w, h) in each frame in real time, where x and y represent the coordinates of the target frame center point, and w and h represent the target frame width and height.

[0054] When the target is initialized, the target category k is obtained through target detection technology. Usually in digital tower applications, k belongs to the set {people, cars, airplanes}.

[0055] The target detection and target tracking methods may adopt any suitable existing connection technology, for example, existing mainstream methods such as the YOLO algorithm and the twin network algorithm.

[0056] 3. PTZ control of ball machine

[0057] Set up the ball camera PTZ control module to estimate the target angular velocity according to the target category, position and size, and use the speed function obtained by offline calibration (the ω(l) function obtained by fitting) to implement the ball camera PTZ control, and adaptively adjust the ball camera's steering, speed and focal length according to the current tracking target information (x, y, w, h; k). Specifically include:

[0058] 1) Steering setting and control: Figure 2 As shown, set the threshold w nin 、w max 、h min 、h max , divide the image into nine areas (similar to the distribution of the nine-square grid), and determine the rotation direction of the ball camera according to the area (specific area) where the target center point (x, y) is located, that is, to rotate in one of the eight directions of up, down, left, right, upper left, upper right, lower left, and lower right, or stop rotating (not rotating). If the rotation direction determined by the current frame is the same as that of the previous frame, the rotation state of the ball camera will not be changed; otherwise, the current rotation will be stopped first, and then the ball camera will be rotated in the new direction. Among them, the threshold can be set according to experience, such as w min 、w max are set to 0.4 and 0.6 times the image width respectively; h nin 、h max Set to 0.4 and 0.6 times the image height respectively.

[0059] 2) Speed ​​setting and control: While setting the rotation direction of the ball camera, estimate the target angular velocity to set the speed of the ball camera. The method is as follows:

[0060] i) Estimate target distance: Based on the current target information (x, y, w, h; k), record the target size where w img 、h img is the image width and height. Given that the current zoom factor of the ball camera is Z, using the pinhole imaging principle (under corresponding settings), the distance d from the target to the ball camera can be estimated as follows:

[0061]

[0062] Where d0 is the reference distance, z0 is the reference zoom factor of the dome camera, and S0 is the size of the target in the image when the distance from the target to the dome camera is d0 and the zoom factor of the dome camera is z0. For targets of different categories k (people, cars, airplanes, etc.), S0 can be pre-determined after d0 and z0 are given. For example, when the zoom factor is 1 (z0=1) and the target distance is 40 meters (d0=40), the size of a pedestrian in a dome camera image (1920*1080) is about 35*75 pixels, then

[0063] ii) Estimating target angular velocity: The estimation of the target angular velocity may be performed based on any suitable target tracking / target velocity (or angular velocity) estimation technique.

[0064] In order to reduce the amount of data processing, as a preferred implementation, based on the current target category k and according to prior knowledge, the target movement speed v can be set for different targets respectively. k For example, in digital tower applications, the speed of people, cars, and airplanes can generally be set to: v 人 =1.5 m / s, v 车 =6 m / s, v 飞机 =70 m / s.

[0065] Assuming that the target moves roughly tangentially (perpendicular to the line of sight), using the aforementioned target distance estimate d, the target angular velocity (degrees / second) can be estimated as:

[0066]

[0067] In practice, the target's moving direction can be determined based on target recognition (target posture). In the case where the target's moving direction is roughly the line of sight, the angular velocity of the target's moving direction can be ignored.

[0068] For the intermediate state between the above two situations, the av calculated above can be given a tangential projection coefficient r t (0 <r t <1), with r t*av is the angular velocity of the target in the intermediate state, but its maximum value does not exceed av. For simplicity, in practice, the speed setting of the ball camera can also be implemented according to the upper limit of the target angular velocity av.

[0069] iii) Set the speed of the ball camera: Substitute the estimated target angular velocity av into the left side of the equal sign of formula 1, and use the root-finding formula of the quadratic equation to solve l (only take the positive root), where a, b, and c are the coefficients obtained when fitting the speed function:

[0070]

[0071] Set the speed level of the ball machine to the upper integer of l

[0072] 3) Setting and controlling the focal length: When the tracking target is located in the stop area in the center of the image, adjust the focal length of the ball camera according to the size of the target. Set the threshold S of the size of the target in the image. min and S max , if the size S of the target is less than S min , then increase the zoom factor Z of the ball camera; if the size S of the target is greater than S max , then reduce Z. Among them, the threshold S min , S max They can be set based on experience, such as 0.04 and 0.08 respectively. The zoom factor Z can be increased or decreased by 0.5 each time.

[0073] Through the above method, the steering, rotation speed and focal length of the ball camera can be dynamically adjusted according to the target position, size and category to better adapt to the tracking / monitoring of moving targets.

[0074] Unless otherwise specified or when one preferred or optional technical means is a further limitation of another technical means, the preferred and optional technical means disclosed in the present invention can be arbitrarily combined to form several different specific implementation methods.

Claims

1. A PTZ target tracking method suitable for a digital tower optical system, characterized in that Access the ball camera video, perform single target tracking according to the initial position of the target, obtain the position and size (x, y, w, h) of the target in each frame of the image in real time, obtain the target category k through target detection technology, and implement PTZ control of the ball camera according to the target information (x, y, w, h; k). In the PTZ control of the ball camera, estimate the target angular velocity according to the target information (x, y, w, h; k), and determine the rotation speed level of the ball camera based on the estimated target angular velocity, where x and y are the coordinates of the center point of the target frame, respectively, and w and h are the width and height of the target frame, respectively.

2. The PTZ target tracking method according to claim 1, characterized in that The method of determining the rotation speed level of the ball camera based on the estimated target motion angular velocity is as follows: based on the ball camera rotation speed function obtained by offline calibration, the rotation speed level variable l of the ball camera under the current target motion angular velocity is calculated, and the rotation speed level of the ball camera is set to the upper integer of l. Or round down The speed function of the ball machine is: ω(l)=to 2 +bl+c, Where ω(l) is the speed of the ball camera, which is taken as the angular velocity of the target motion, and a, b, and c are the corresponding fitting polynomial coefficients obtained by offline calibration.

3. The PTZ target tracking method according to claim 1, characterized in that The method for estimating the target angular velocity is: presetting a reference state of the target category, measuring the size of the target in the image under the reference state, and calculating the real-time distance from the target to the ball camera based on the following formula: The target motion angular velocity is calculated based on the following formula: In the formula d0, S0 and z0 are the distance from the target to the ball camera, the size of the target in the image and the zoom factor of the ball camera in the reference state, respectively. d, S and z are the distance from the target to the ball camera, the size of the target in the image and the zoom factor of the ball camera in the current state, respectively. w img and h img are the width and height of the image, w0 and h are the width and height of the target in the image under the reference state, w and h are the width and height of the target in the image under the current state, av is the angular velocity of the target, v k Preset values ​​for motion speed for corresponding target categories.

4. The PTZ target tracking method according to any one of claims 1 to 3, characterized in that The image is divided into nine areas vertically and horizontally, and the rotation direction of the ball camera and / or the zoom multiple of the ball camera are determined based on the area where the target is located.

5. The PTZ target tracking method according to claim 4, characterized in that The upper and lower thresholds w of the central area in the image width direction of the nine areas min and w max The upper and lower thresholds h in the image height direction are set to 0.4 and 0.6 times the image width respectively. min and h max Set to 0.4 and 0.6 times the image height respectively.

6. The PTZ target tracking method according to claim 4, characterized in that The method of determining the rotation direction of the ball camera based on the area where the target is located is: when the target is in any of the eight areas of the image, namely, upper, lower, left, right, upper left, upper right, lower left and lower right, the direction of the ball camera is determined according to the direction of the area where the target is located relative to the central area, so that the target in the image moves from the area where it is located toward the central area. When the target is in the central area of ​​the image, the ball camera is not rotated.

7. The PTZ target tracking method according to claim 4, characterized in that The method of determining the zoom multiple of the ball camera based on the area where the target is located is: when the target is in the central area of ​​the image, the focal length of the ball camera is determined or adjusted according to the target size, so that the target size in the image is maintained at or adjusted to a set target size range.

8. The PTZ target tracking method according to claim 7, characterized in that The upper and lower thresholds S of the target size range are set min and S max 0.04 and 0.08 respectively.

9. The PTZ target tracking method according to claim 8, characterized in that If the size of the target S is less than S min , then increase the zoom factor Z of the ball camera; if the size S of the target is greater than S max , then reduce the zoom factor Z of the ball camera, and the zoom factor Z increases or decreases step by step, each time increasing or decreasing by 0.

5.

10. A PTZ target tracking system suitable for a digital tower optical system, characterized in that The PTZ target tracking method suitable for a digital tower optical system according to any one of claims 1 to 9 is used to implement PTZ control of a ball camera, comprising: The ball camera speed measurement calibration module is used to establish the functional relationship between the speed level and angular velocity of the ball camera; The target detection and tracking module is used to track the specified target in the ball camera video, identify the target category, and obtain the target position and size in real time; The ball camera PTZ control module is used to estimate the target motion angular velocity according to the target type, position and size, and implement the ball camera PTZ control using the speed function obtained by offline calibration to set the speed, steering and focal length of the ball camera.

Images