Holder tracking method and device, equipment and storage medium

Abstract

The invention discloses a holder tracking method, device and equipment and a storage medium, and the method comprises the steps: obtaining a plurality of frames of first images collected by a collection device at different moments, each frame of first image containing the same target, and the collection device is disposed on a holder; based on a first holder coordinate corresponding to the target in each frame of first image, respectively determining a real moving speed of the target corresponding to each frame of first image; in response to the condition that the holder enters a target tracking state, predicting a first predicted moving speed of the target at the first control moment, the first predicted moving speed being obtained by predicting based on each real moving speed; and taking the first predicted moving speed as the moving speed control quantity of the holder at the first control moment. Through the above mode, the method can achieve the quick and precise control of the holder, and can achieve the effective tracking of a target.

Description

Technical Field

[0001] The present application relates to the field of camera technology, and in particular to a pan / tilt tracking method, device and storage medium. Background Art

[0002] In the fields of video surveillance, photography tracking, drone tracking, etc., accurate tracking of dynamic targets is a key technology. As a device that can flexibly adjust the viewing angle, the gimbal is widely used in the tracking of dynamic targets. Driven by a built-in motor, the gimbal can achieve multi-directional movement such as horizontal, vertical and even rotation, thereby effectively tracking the target. However, the overall weight of the existing gimbal is relatively large, especially when the gimbal is first controlled from static to moving, the movement of the gimbal is prone to delay. When the target moves faster, it is easy to lose the target. How to control the rotation of the gimbal in time to achieve effective tracking of the target is of great significance. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a pan-tilt tracking method, device, equipment and storage medium, which can realize early and accurate control of the pan-tilt, and thus realize effective tracking of the target.

[0004] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a gimbal tracking method, the method comprising: obtaining multiple frames of first images collected by an acquisition device at different times, each frame of the first image contains the same target, wherein the acquisition device is arranged on the gimbal; based on the first gimbal coordinates corresponding to the target in each frame of the first image, determining the actual movement speed of the target corresponding to each frame of the first image; in response to the gimbal entering a tracking state of the target, predicting a first predicted movement speed of the target at the first control moment, wherein the first predicted movement speed is obtained based on the prediction of each actual movement speed; and using the first predicted movement speed as the movement speed control amount of the gimbal at the first control moment.

[0005] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide a pan-tilt tracking device, including: an acquisition module, a determination module, a prediction module and a control amount determination module. The acquisition module is used to acquire multiple frames of first images acquired by the acquisition device at different times, each frame of the first image contains the same target, wherein the acquisition device is arranged on the pan-tilt; the determination module is used to determine the actual moving speed of the target corresponding to each frame of the first image based on the first pan-tilt coordinates corresponding to the target in each frame of the first image; the prediction module is used to predict the first predicted moving speed of the target at the first control moment in response to the pan-tilt entering the tracking state of the target, wherein the first predicted moving speed is obtained based on the prediction of each actual moving speed; the control amount determination module is used to use the first predicted moving speed as the moving speed control amount of the pan-tilt at the first control moment.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0007] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium for storing program instructions, which can be executed to implement the above method.

[0008] The above scheme first uses the first gimbal coordinates corresponding to the target in the first image of each frame to determine the real moving speed of the target in the first image of each frame respectively, and then predicts the first predicted moving speed of the target at the first control moment of the gimbal when the gimbal enters the tracking state of the target, and uses the first predicted moving speed as the moving speed control amount of the gimbal at the first control moment. It can be seen that when the gimbal enters the tracking state of the target, the present application uses the predicted moving speed of the target as the moving speed control amount of the gimbal at the first control moment. Compared with the method of controlling by using the instant speed, the present application can control the gimbal to move to the position according to the predicted moving speed control amount in advance at the first control moment, so that the target can be quickly tracked when the gimbal is controlled for the first time. Furthermore, the present application uses the predicted moving speed of the target as the moving speed control amount of the gimbal, which can accurately match the moving speed of the target, so that the target can be accurately tracked. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of an embodiment of a pan / tilt tracking method provided by the present application;

[0010] Figure 2 yes Figure 1 The flowchart of step S13 is shown as an embodiment;

[0011] Figure 3 yes Figure 2 The flowchart of step S21 of an embodiment is shown;

[0012] Figure 4 It is a schematic diagram of the framework of an embodiment of a pan-tilt tracking device provided by the present application;

[0013] Figure 5 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0014] Figure 6 It is a schematic diagram of the framework of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0016] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0017] See also Figure 1 , Figure 1 is a flow chart of an embodiment of the pan-tilt tracking method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0018] S11: Acquire multiple frames of first images acquired by an acquisition device at different times, each frame of the first image contains the same target, wherein the acquisition device is disposed on a pan-tilt platform.

[0019] This embodiment is used to determine in advance the movement speed control amount of the gimbal at the first control moment, so that when the gimbal is controlled for the first time, the gimbal can be controlled in advance to move with the movement speed control amount, and then can move to the corresponding position in a timely and accurate manner to track the target.

[0020] The PTZ is a device that can carry and flexibly adjust the viewing angle. The acquisition device on the PTZ can move in multiple directions such as horizontal, vertical and even rotation to capture the images or scenes that need to be recorded in the monitoring area.

[0021] The acquisition device may be any device capable of acquiring images, and may be, but is not limited to, a visible light camera, an infrared or thermal imager, etc. The first image may be, for example, a visible light image, an infrared image, or a thermal imaging image.

[0022] In this embodiment, each first image frame contains the same target, which may be any object that needs to be monitored or tracked, such as a person or vehicle in the monitoring area, or any movable object. The target may be specifically defined according to the actual application scenario.

[0023] It should be noted that, in this embodiment, the multiple frames of first images collected at different times are used to determine the gimbal coordinates (first gimbal coordinates) of the target in the gimbal coordinate system at different times, and then the actual moving speed of the target at each time is determined based on the gimbal coordinates of the target at each time, and then the predicted moving speed of the target at each future time (including the first control time) is predicted based on the actual moving speed of the target at each time, so as to control the gimbal to track the target through the predicted moving speed of the target.

[0024] S12: Based on the first gimbal coordinates corresponding to the target in each frame of the first image, respectively determine the actual moving speed of the target corresponding to each frame of the first image.

[0025] In one embodiment, the first gimbal coordinates corresponding to the target in each frame of the first image correspond to the first gimbal coordinates of the target at each moment (image acquisition moment or image detection completion moment). The first gimbal coordinates of the target at each moment are determined by performing target detection on the corresponding first image acquired at each moment using a relevant target detection model or a target tracking model.

[0026] In one embodiment, the step of obtaining the first gimbal coordinates corresponding to the target in each frame of the first image includes: performing target detection on each frame of the first image in sequence to obtain the image coordinates of the target frame of the target in each frame of the first image; for each first image, converting the image coordinates of the target frame corresponding to the first image to the gimbal coordinate system to obtain the second gimbal coordinates; using the second gimbal coordinates of the target frame, calculating the gimbal coordinates of a preset position point in the target frame as the first gimbal coordinates of the target corresponding to the first image.

[0027] The purpose of target detection is to identify specific targets (such as people, cars, animals, etc.) in an image and return the positions of these targets in the image. The position of the target is usually represented in the form of a target frame, which contains the image coordinates of the target. The image coordinates of the target frame are, for example, the coordinates of the upper left and lower right points of the target frame, or the coordinates of the upper left, lower right and center points of the target frame.

[0028] In this embodiment, the image coordinates of the target are first converted into coordinates in the pan-tilt coordinate system, namely, the second pan-tilt coordinates. The converted second pan-tilt coordinates represent the position of the target in the pan-tilt coordinate system.

[0029] In order to facilitate the subsequent determination of the target's real moving speed at each moment based on the target's gimbal coordinates at each moment, a preset position point for representing the target position can be found in the target frame, and the gimbal coordinates of the preset position point are used as the first gimbal coordinates of the target corresponding to the first image, that is, the first gimbal coordinates represent the coordinates of the preset position point of the target in the gimbal coordinate system. The preset position point can be any position point in the target frame, preferably the center point of the target frame.

[0030] In one application scenario, in order to achieve reliable all-weather monitoring, a collection device including a visible light camera and a thermal imaging camera may be provided to collect visible light images and thermal imaging images. That is, each frame of the first image collected in this implementation scenario includes two modal images, a visible light image and a thermal imaging image, and the first gimbal coordinates are determined based on the target detection results of the two modal images.

[0031] Specifically, target detection can be performed on visible light images and thermal imaging images respectively to obtain the image coordinates of the target frame in each image, and then for each image, the image coordinates of the target frame in the corresponding image are converted into second gimbal coordinates in the gimbal coordinate system, and finally, the gimbal coordinates of a preset position point (such as the center point) in the target frame are used as the first gimbal coordinates of the image corresponding to the target.

[0032] It should be noted that when the gimbal enters the target tracking state (when controlling the gimbal to track the target), it can distinguish between day and night. During the day, the detection results of the visible light image are used to control the gimbal to track the target, and at night, the detection results of the thermal imaging image are used to control the gimbal to track the target. Of course, in other application scenarios, cameras of other modes can be set to collect corresponding images according to the needs of the scenario, or only one mode of camera can be set.

[0033] In this embodiment, determining the actual moving speed of the target corresponding to the first image of each frame includes: taking the first image as the current image, obtaining the first coordinate difference between the first gimbal coordinate corresponding to the current image and the first gimbal coordinate corresponding to the previous first image, and obtaining the target detection time of the current image; then obtaining the ratio between the first coordinate difference and the target detection time as the actual moving speed of the target corresponding to the current image.

[0034] For example, please refer to the following formula:

[0035] pv(i+1)=[p(i+1)-p(i)] / cost(i)

[0036] tv(i+1)=[t(i+1)-t(i)] / cost(i)

[0037] Wherein, (p(i+1), t(i+1)) is the first gimbal coordinate corresponding to the current image, (p(i), t(i)) is the first gimbal coordinate corresponding to the previous first image, cost(i) represents the target detection time of the current image, pv(i+1) and tv(i+1) represent the actual moving speed of the target corresponding to the current image in two different directions (horizontal and vertical directions).

[0038] The target detection time may be the difference between the start time of target detection of the current image and the completion time of target detection, or the difference between the completion time of target detection of the current image and the completion time of target detection of the previous first image.

[0039] S13: In response to the gimbal entering a tracking state of the target, predicting a first predicted moving speed of the target at the first control moment, wherein the first predicted moving speed is obtained based on predictions of the actual moving speeds.

[0040] In the entire application, the gimbal includes at least two states, one is the dormant state where the target does not need to be tracked, and the other is the tracking state where the target needs to be tracked. The gimbal can perform image acquisition, target detection, and detection result storage in the dormant state. Generally speaking, the gimbal in the dormant state is stationary. The gimbal enters the tracking state, which means the gimbal is controlled to follow the target.

[0041] In one implementation scenario, whether the gimbal enters a state of tracking a target is determined based on whether there is a target that triggers an alarm. If there is a target that triggers an alarm, the gimbal is determined to enter a state of tracking the target. In this case, the gimbal can be controlled to move to track the target.

[0042] It should be noted that the target area includes the alarm area, and the target entering the monitoring area does not mean that the target enters the alarm area. In an application scenario, considering that the radar can scan the surrounding environment in all directions and capture information from any direction, while the gimbal can usually only focus on one direction at the same time and cannot monitor multiple directions at the same time, for safety, the target entering the monitoring area can be quickly captured. The radar can first detect whether there is a target in the entire monitoring area, and when a target is detected, a control instruction is sent to the device end where the gimbal is located, so that after the device end where the gimbal is located receives the control instruction sent by the radar, based on the control instruction, the gimbal is controlled to move to the target position, so that the acquisition device set on the gimbal is used to collect images of the target and use the image to track the target, and then analyze whether the tracked target triggers the alarm condition based on the target tracking data at the image level, and when it is determined that the target triggers the alarm condition, the gimbal in a stationary state is controlled to enter the tracking state of the triggered alarm target (alarm target) (that is, the gimbal is controlled to track the alarm target).

[0043] In a specific embodiment, determining that the gimbal enters a state of tracking a target includes: receiving a control instruction sent by a radar, wherein the control instruction includes a control amount for the gimbal, then controlling the gimbal to move according to the control amount, controlling the gimbal to be stationary, and acquiring a second image acquired by an acquisition device after the gimbal is stationary, then acquiring the first gimbal coordinates and the real moving speed corresponding to the target in the second image; and detecting whether the target in the second image meets an alarm condition; in response to the target in the second image meeting the alarm condition, determining that the gimbal enters a state of tracking the target. The control amount for the gimbal is determined by the radar based on the position of the target in the gimbal coordinate system.

[0044] That is, in this embodiment, after the radar sends a control instruction, it can control the gimbal to move to the target position based on the control amount in the received control instruction, and then control the gimbal to be still. At this time, the device where the gimbal is located collects a second image of the target, and obtains the first gimbal coordinates and the actual moving speed corresponding to the target based on the image detection result, so as to track the target and detect whether the target in the second image meets the alarm conditions. If so, it is determined that the gimbal enters the tracking state of the target (the target that triggers the alarm).

[0045] In some embodiments, a tripwire or alarm area can be set in advance in the monitoring area. If a target is detected entering the tripwire or alarm area, the target is considered to meet the alarm condition and the alarm is triggered. In the presence of a target that triggers the alarm, the pan / tilt head is controlled to rotate to track the target that triggers the alarm.

[0046] Specifically, whether the target triggers an alarm condition can be determined based on the trajectory data of the target determined based on the second image, for example, whether the alarm condition is met by determining whether the target is moving in the direction of a trip wire or an alarm area, whether it enters the alarm area, etc. Specifically, the alarm condition can be pre-set according to the actual scene requirements, and is not specifically limited here.

[0047] It is understandable that in the above scheme, before the radar detects the target and sends the command to the gimbal, the gimbal is stationary, and before the gimbal moves to the target position according to the command and detects the alarm target, the gimbal is also stationary. This scheme takes into account that in the process of detecting targets and alarming targets during the movement of the gimbal, the background targets are complex and prone to false alarms. When the gimbal is controlled to track the target through false alarms, it is easy to mistrack. In order to solve this problem, this scheme first makes the gimbal stationary, and when a moving target is stably detected, target tracking is triggered to reduce false alarms and mistracking.

[0048] In an implementation scenario, when the gimbal is in the tracking state, if any of the following interruption events occurs, the tracking state of the target is terminated, and the control command of the radar is continuously monitored. The interruption events include any of the following: the duration of the tracking state reaches the upper limit of the tracking state, a new control command of the radar is received, a rotation command of the gimbal is received, and the target tracking fails.

[0049] It should be noted that, in step S13, the first predicted moving speed is predicted based on the actual moving speeds of the target.

[0050] In one embodiment, the speed prediction formula for the target is first obtained by fitting each real moving speed, and then the speed prediction formula is used to predict the first predicted moving speed of the target at the first control moment. The independent variable in the speed prediction formula represents the moment corresponding to each frame image, or represents the sequence number of each frame image sorted in the order of the corresponding moment, and the dependent variable in the speed prediction formula is the predicted moving speed of the image corresponding to the target. The speed prediction formula for the target can be obtained by fitting using the least squares method, as described below; of course, other existing methods can also be used to fit the speed prediction formula.

[0051] In an application scenario, such as a scene where the target moves slowly or the target size is relatively large, the first control moment or the serial number of the image corresponding to the first control moment can be directly input into the speed prediction formula to predict the first predicted moving speed of the target at the first control moment.

[0052] The speed prediction formula obtained by fitting can refer to the following formula:

[0053] pv_{n}=a0+a1*n+a2*n^{2}+……+bk*n^{k}

[0054] tv_{n}=b0+b1*n+b2*n^{2}+……+bk*n^{k}

[0055] Where n represents each time point in the future, or the serial number of each image frame, and pv_{n} and tv_{n} represent the speed of the target in different directions at the next n moments.

[0056] In this application scenario, the first control moment or the serial number of the image corresponding to the first control moment can be directly input into the above speed prediction formula to obtain the first predicted moving speed of the target at the first control moment.

[0057] It should be noted that in one implementation scenario, since the target tracked by the pan / tilt is the target that triggers the alarm, the pan / tilt enters the tracking state of the target when the target satisfies the alarm condition (triggering the alarm), that is, the target that triggers the alarm needs to be determined first, and then the pan / tilt is controlled to track the target that triggers the alarm. In order to distinguish the image of the target when the alarm is not triggered and when the alarm is triggered, the first image that triggers the alarm based on target detection can be called the second image, and the real moving speed of the target corresponding to the second image is obtained, and then based on the real moving speed of the target corresponding to the second image, the first predicted moving speed of the target at the first control moment is predicted.

[0058] Furthermore, considering that there will be a time interval from the moment when the target detection or alarm is triggered in the second image to the moment when the pan-tilt movement is first controlled, such as the time taken from the target detection to the time when the alarm is triggered after the alarm detection, and the time taken from the time when the alarm is triggered to the time when the pan-tilt movement is controlled (such as the time taken for the logical conversion from the radar to the pan-tilt control), if the time is not taken into account, and the predicted speed of the target corresponding to the next frame image based on the actual moving speed of the target corresponding to the moment when the target detection is completed in the second image is directly used as the first predicted speed at the first control moment, then due to the speed change within the time interval (time consumption), it is easy to cause the predicted first predicted moving speed of the target at the first control moment to be inaccurate. Therefore, in order to accurately predict the first predicted moving speed at the first control moment, the duration between the moment when the target detection of the second image is completed or the moment when the alarm condition is met and the moment when the first control is performed can be obtained, and based on the actual moving speed of the target corresponding to the second image, the speed increase of the target during this time period is determined, and the sum of the actual moving speed of the target corresponding to the current frame image and the moving speed increase during this time period is used as the first predicted moving speed of the target corresponding to the next frame image (first control moment). The specific method of predicting the first predicted moving speed of the target at the first control moment can be referred to. Figure 2 Description of the illustrated embodiment.

[0059] S14: Using the first predicted moving speed as the moving speed control amount of the pan / tilt head at the first control moment.

[0060] In this embodiment, after determining the first predicted moving speed of the target at the first control moment, the first predicted moving speed is directly used as the moving speed control amount of the gimbal at the first control moment, so that when the gimbal moves according to the moving speed control amount, the gimbal can quickly track the target.

[0061] Since the first predicted moving speed is a pre-predicted moving speed control amount for the pan / tilt platform at the first control moment, the pan / tilt platform can start moving at the first predicted moving speed in advance, thereby enabling the pan / tilt platform to move in advance and quickly track the target.

[0062] In one implementation scenario, the above-mentioned first gimbal coordinates are the gimbal coordinates corresponding to the target in the gimbal coordinate system that need to be sent to the gimbal when the preset position point of the target box in the corresponding image is centered to the center of the image through absolute positioning. In this way, the target appears in the middle of the captured image during the process of the gimbal tracking the target.

[0063] The above scheme first uses the first gimbal coordinates corresponding to the target in the first image of each frame to determine the real moving speed of the target in the first image of each frame respectively, and then predicts the first predicted moving speed of the target at the first control moment of the gimbal when the gimbal enters the tracking state of the target, and uses the first predicted moving speed as the moving speed control amount of the gimbal at the first control moment. It can be seen that when the gimbal enters the tracking state of the target, the present application uses the predicted moving speed of the target as the moving speed control amount of the gimbal at the first control moment. Compared with the method of controlling by using the instant speed, the present application can control the gimbal to move to the position according to the predicted moving speed control amount in advance at the first control moment, so that the target can be quickly tracked when the gimbal is controlled for the first time. Furthermore, the present application uses the predicted moving speed of the target as the moving speed control amount of the gimbal, which can accurately match the moving speed of the target, so that the target can be accurately tracked.

[0064] See also Figure 2 , Figure 2 yes Figure 1 The flowchart of step S13 of an embodiment is shown. In this embodiment, predicting the first predicted moving speed of the target at the first control moment includes:

[0065] S21: Obtain the increase in the target's moving speed in the first time period, wherein the first time period is the time period from the moment when target detection in the second image is completed to the moment when the target is first controlled, or the time period from the moment when it is determined that the target in the second image meets the alarm condition to the moment when the target is first controlled, and the increase in the moving speed is obtained based on the prediction of each real moving speed.

[0066] See also Figure 3 , Figure 3 yes Figure 2 The flowchart of step S21 of an embodiment is shown as follows. In this embodiment, obtaining the increase in the moving speed of the target during the first time period includes:

[0067] S31: Acquire a second predicted moving speed of the target corresponding to a third image, where the second predicted moving speed is predicted based on each actual moving speed, and the third image is captured by a capture device and is the next image after the second image.

[0068] S32: taking the difference between the second predicted moving speed and the actual moving speed corresponding to the second image as the acceleration corresponding to the third image.

[0069] To facilitate understanding of step S31 and step S32, the following formula may be referred to:

[0070] a(pvn)=△pv=pv_{n}-pv_{n-1}

[0071] Wherein, pv_{n-1} represents the actual moving speed corresponding to the second image, pv_{n} represents the second predicted moving speed corresponding to the third image (the image after the second image), and a(pvn) represents the acceleration of the third image (used to characterize the speed change from the second image to the third image).

[0072] The second predicted moving speed is predicted based on each actual moving speed.

[0073] In a specific embodiment, the second predicted moving speed is calculated using the speed prediction formula obtained by fitting based on the actual moving speeds.

[0074] S33: The product of the acceleration and the first duration is taken as the increase in the moving speed.

[0075] S22: Obtain the sum of the actual moving speed corresponding to the second image and the moving speed increase as the first predicted moving speed.

[0076] For example, please refer to the following formula:

[0077] pv=pv_{n-1}+a(pvn)*(T1-T0)

[0078] Where pv is the first predicted moving speed, pv_{n-1} is the actual moving speed corresponding to the second image, T1 represents the first control time, T0 represents the time when the target detection of the second image is completed or the time when the target in the second image meets the alarm condition, (T1-T0) represents the first duration, and a(pvn)*(T1-T0) represents the increase in moving speed.

[0079] It should be noted that the above mainly shows how to determine the movement speed control amount of the pan-tilt at the first control moment (the first tracking frame when the pan-tilt changes from stationary to rotating). It can be understood that the pan-tilt needs to continue tracking the target until the target leaves the alarm area or the monitoring area. Therefore, after determining the movement speed control amount of the pan-tilt at the first control moment, it is also necessary to determine the movement speed control amount of the pan-tilt at subsequent moments.

[0080] Specifically, after the predicted moving speed is used as the moving speed control amount of the pan / tilt at the first control moment, the following steps are also included:

[0081] First, obtain a second coordinate difference between the third gimbal coordinate and the first gimbal coordinate of the target in the fourth image, wherein the fourth image is acquired by the acquisition device and is the image that has most recently completed target detection, the third gimbal coordinate represents the gimbal coordinate of the target in the fifth image, the fifth image is the next frame image of the fourth image, and the third gimbal coordinate is calculated based on the first gimbal coordinate corresponding to the fourth image and the predicted moving speed of the target corresponding to the fifth image.

[0082] Second, the ratio of the second coordinate difference to the time difference between the fourth image and the fifth image is used as the movement speed control amount of the pan / tilt head at a non-first control moment.

[0083] Exemplarily, the control amount of the moving speed of the PTZ at a non-first control moment can be expressed by the following formula:

[0084] [p(n)-p(n-1)] / △t

[0085] Wherein, p(n) is the third gimbal coordinate of the target in the fifth image, which is calculated based on the first gimbal coordinate corresponding to the previous frame image (fourth image) of the fifth image and the predicted moving speed of the target corresponding to the fifth image. p(n-1) represents the first gimbal coordinate of the target in the fourth image. △t represents the time difference between the fourth image and the fifth image, which can be the time difference between the predicted moment of the fifth image and the acquisition moment of the fourth image, or the time difference between the predicted moment of the fifth image and the moment when the target detection of the fourth image is completed.

[0086] Optionally, the third gimbal coordinates of the target in the fifth image can be calculated based on the first gimbal coordinates corresponding to the previous frame image and the predicted moving speed of the fifth image. The predicted moving speed of the fifth image can be predicted using the speed prediction formula obtained by fitting, or can be predicted using the above-mentioned moving speed increase amount.

[0087] Of course, in other embodiments, the third gimbal coordinates of the target in the fifth image may also be predicted using existing prediction methods. For example, the third gimbal coordinates of the target in the fifth image may be predicted using a related model based on the first gimbal coordinates corresponding to the first several frames of the fifth image; or, for example, the first gimbal coordinates corresponding to the first several frames of the fifth image may be fitted to obtain a position prediction equation, and then the third gimbal coordinates of the target in the fifth image may be predicted using the position prediction equation.

[0088] In one embodiment, the speed prediction formula may be obtained by using the following least squares fitting method:

[0089] First, assume that the fitting polynomial of the velocity is:

[0090] y=a0+a1x+...+a k xk

[0091] Then, the sum of the distances from the target's actual moving speed at different times to the speed curve represented by the fitting polynomial is determined, that is, the sum of squared deviations is as follows:

[0092]

[0093] Among them, in order to obtain the a value that meets the conditions, the partial derivative of ai is calculated on the right side of the equation, and we get:

[0094]

[0095] Then, simplify the left side of the equation and you should get the following equation:

[0096]

[0097] Representing these equations in matrix form, we get the following matrix:

[0098]

[0099] Simplifying this Vandermonde matrix yields:

[0100]

[0101] Among them, a0...ak are the coefficients of the fitting polynomial. Generally, the power term k can be taken as 2 in order to have a certain generalization ability and avoid overfitting.

[0102] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of an embodiment of a pan-tilt tracking device provided by the present application. In this embodiment, the pan-tilt tracking device 40 includes an acquisition module 41, a determination module 42, a prediction module 43 and a control amount determination module 44. The acquisition module 41 is used to acquire multiple frames of first images acquired by the acquisition device at different times, each frame of the first image contains the same target, wherein the acquisition device is arranged on the pan-tilt; the determination module 42 is used to determine the actual moving speed of the target corresponding to each frame of the first image based on the first pan-tilt coordinates corresponding to the target in each frame of the first image; the prediction module 43 is used to predict the first predicted moving speed of the target at the first control moment in response to the pan-tilt entering the tracking state of the target, wherein the first predicted moving speed is obtained based on the prediction of each actual moving speed; the control amount determination module 44 is used to use the first predicted moving speed as the moving speed control amount of the pan-tilt at the first control moment.

[0103] In some embodiments, before the prediction module 43 predicts the first predicted moving speed of the target at the first control moment, it also includes: obtaining the actual moving speed of the target corresponding to the second image, the second image is acquired by the acquisition device, and the pan / tilt enters the tracking state of the target is triggered by the target in the second image meeting the alarm condition; predicting the first predicted moving speed of the target at the first control moment, including: obtaining the increase in the moving speed of the target in the first time length, wherein the first time length is the time length from the moment when the target detection in the second image is completed to the moment of first control, or the time length from the moment when the target in the second image meets the alarm condition to the moment of first control, and the increase in moving speed is obtained based on the prediction of each actual moving speed; obtaining the sum of the actual moving speed corresponding to the second image and the increase in moving speed as the first predicted moving speed.

[0104] In some embodiments, obtaining the increase in the moving speed of the target in a first time period includes: obtaining a second predicted moving speed of the target corresponding to a third image, the second predicted moving speed being predicted based on each actual moving speed, the third image being an image captured by a capture device and being the subsequent image of the second image; taking the difference between the second predicted moving speed and the actual moving speed corresponding to the second image as the acceleration corresponding to the third image; and taking the product of the acceleration and the first time period as the increase in the moving speed.

[0105] In some embodiments, obtaining a second predicted moving speed of the target corresponding to the third image includes: using a speed prediction formula of the target to calculate the second predicted moving speed, wherein the speed prediction formula is obtained by fitting various real moving speeds.

[0106] In some embodiments, before the prediction module 43 predicts the first predicted moving speed of the target at the first control moment in response to the gimbal entering a tracking state of the target, it also includes: using each real moving speed to fit a speed prediction formula for the target, the independent variable of the speed prediction formula represents the moment corresponding to each frame image, and the dependent variable is the predicted moving speed of the target corresponding image; predicting the first predicted moving speed of the target at the first control moment includes: using the speed prediction formula to predict the first predicted moving speed of the target at the first control moment.

[0107] In some embodiments, the first gimbal coordinates are obtained by performing target detection on the corresponding image; based on the first gimbal coordinates corresponding to the target in each frame of the first image, the actual movement speed of the target corresponding to each frame of the first image is determined, including: taking the first image as the current image, obtaining the first coordinate difference between the first gimbal coordinates corresponding to the current image and the first gimbal coordinates corresponding to the previous first image, and obtaining the target detection time of the current image; obtaining the ratio between the first coordinate difference and the target detection time as the actual movement speed of the target corresponding to the current image.

[0108] In some embodiments, the first gimbal coordinates represent the coordinates of a preset position point of the target in the gimbal coordinate system; before the determination module 42 determines the actual moving speed of the target corresponding to each frame of the first image based on the first gimbal coordinates corresponding to the target in each frame of the first image, it also includes: performing target detection on each frame of the first image in sequence to obtain the image coordinates of the target frame of the target in each frame of the first image; for each first image, converting the image coordinates of the target frame corresponding to the first image to the gimbal coordinate system to obtain the second gimbal coordinates; using the second gimbal coordinates of the target frame, calculating the gimbal coordinates of the preset position point in the target frame as the first gimbal coordinates of the target corresponding to the first image.

[0109] In some embodiments, target detection is achieved using a target detection model or a target tracking model; and / or, each frame of the first image includes a visible light image and a thermal imaging image, and the first gimbal coordinates are determined based on the target detection result of the visible light image and the target detection result of the corresponding thermal imaging image.

[0110] In some embodiments, the first gimbal coordinates are obtained by performing target detection on the corresponding image; after the control amount determination module 44 uses the first predicted moving speed as the moving speed control amount of the gimbal at the first control moment, it also includes: obtaining the second coordinate difference between the third gimbal coordinates and the first gimbal coordinates of the target in the fourth image, wherein the fourth image is acquired by the acquisition device and is the image that has recently completed target detection, the third gimbal coordinates represent the gimbal coordinates of the target in the fifth image, the fifth image is the next frame image of the fourth image, and the third gimbal coordinates are calculated based on the first gimbal coordinates corresponding to the fourth image and the predicted moving speed of the target corresponding to the fifth image; the ratio of the second coordinate difference to the time difference between the fourth image and the fifth image is used as the moving speed control amount of the gimbal at a non-first control moment.

[0111] In some embodiments, before the prediction module 43 responds to the gimbal entering a tracking state for a target and predicts the first predicted moving speed of the target at the first control moment, it also includes: receiving a control instruction issued by the radar, the control instruction including a control amount for the gimbal; after controlling the gimbal to move according to the control amount, controlling the gimbal to be still; acquiring a second image acquired by the acquisition device after the gimbal is still; acquiring the first gimbal coordinates and actual moving speed corresponding to the target in the second image; and, detecting whether the target in the second image meets the alarm condition; in response to the target in the second image meeting the alarm condition, determining that the gimbal enters a tracking state for the target.

[0112] In some embodiments, it also includes: when the gimbal is in a tracking state, in response to the current existence of an interruption event, the tracking state is ended, and whether a control instruction of the radar is received is monitored, wherein the interruption event includes any one of the following: the duration of the tracking state reaches the tracking upper limit value, a new control instruction of the radar is received, a rotation instruction of the gimbal is received, and the target tracking fails.

[0113] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 50 includes a memory 51 and a processor 52 coupled to each other.

[0114] The memory 51 stores program instructions, and the processor 52 is used to execute the program instructions stored in the memory 51 to implement the steps of any of the above method implementations. In a specific implementation scenario, the electronic device 50 may include, but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0115] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps of any of the above-mentioned embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0116] See also Figure 6 , Figure 6It is a schematic diagram of the framework of the computer-readable storage medium provided by the present application. The computer-readable storage medium 60 of the embodiment of the present application stores a program instruction 61, and when the program instruction 61 is executed, the method provided by any embodiment of the above method and any non-conflicting combination is implemented. Among them, the program instruction 61 can form a program file and be stored in the above-mentioned computer-readable storage medium 60 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 60 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0117] The above scheme first uses the first gimbal coordinates corresponding to the target in the first image of each frame to determine the real moving speed of the target in the first image of each frame respectively, and then predicts the first predicted moving speed of the target at the first control moment of the gimbal when the gimbal enters the tracking state of the target, and uses the first predicted moving speed as the moving speed control amount of the gimbal at the first control moment. It can be seen that when the gimbal enters the tracking state of the target, the present application uses the predicted moving speed of the target as the moving speed control amount of the gimbal at the first control moment. Compared with the method of controlling by using the instant speed, the present application can control the gimbal to move to the position according to the predicted moving speed control amount in advance at the first control moment, so that the target can be quickly tracked when the gimbal is controlled for the first time. Furthermore, the present application uses the predicted moving speed of the target as the moving speed control amount of the gimbal, which can accurately match the moving speed of the target, so that the target can be accurately tracked.

[0118] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0119] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0120] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0124] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A pan / tilt tracking method, characterized in that: include: Acquire multiple frames of first images acquired by an acquisition device at different times, each of the first images containing the same target, wherein the acquisition device is disposed on a pan-tilt platform; Based on the first gimbal coordinates corresponding to the target in each frame of the first image, respectively determine the actual moving speed of the target corresponding to each frame of the first image; In response to the pan / tilt platform entering a tracking state of the target, predicting a first predicted moving speed of the target at a first control moment, wherein the first predicted moving speed is predicted based on each of the real moving speeds; The first predicted moving speed is used as the moving speed control amount of the pan / tilt platform at the first control moment.

2. The method according to claim 1, characterized in that Before predicting the first predicted moving speed of the target at the first control moment, the method further includes: Acquiring a real moving speed of the target corresponding to a second image, where the second image is acquired by the acquisition device, and the pan / tilt head enters a tracking state of the target when the target in the second image meets an alarm condition; The predicting a first predicted moving speed of the target at the first control moment includes: Obtaining an increase in the moving speed of the target in a first time period, wherein the first time period is a time period from a time when target detection of the second image is completed to a time period of the first control, or a time period from a time when the target in the second image is determined to meet an alarm condition to a time period of the first control, and the moving speed increase is obtained based on the prediction of each of the real moving speeds; The sum of the actual moving speed corresponding to the second image and the moving speed increase is obtained as the first predicted moving speed.

3. The method according to claim 2, characterized in that The step of obtaining the increase in the moving speed of the target during the first time period includes: Acquire a second predicted moving speed of the target corresponding to a third image, where the second predicted moving speed is predicted based on each of the actual moving speeds, and the third image is captured by the acquisition device and is an image subsequent to the second image; taking the difference between the second predicted moving speed and the actual moving speed corresponding to the second image as the acceleration corresponding to the third image; The product of the acceleration and the first duration is used as the increase in the moving speed.

4. The method according to claim 3, characterized in that: The obtaining of a second predicted moving speed of the target corresponding to the third image includes: A second predicted moving speed is calculated using a speed prediction formula of the target, wherein the speed prediction formula is obtained by fitting the actual moving speeds.

5. The method according to claim 1, characterized in that: Before predicting a first predicted moving speed of the target at the first control moment in response to the pan / tilt platform entering a tracking state of the target, the method further includes: Using the actual moving speeds, a speed prediction formula of the target is fitted, wherein the independent variable of the speed prediction formula represents the time corresponding to each frame image, and the dependent variable is the predicted moving speed of the target corresponding to the image; The predicting a first predicted moving speed of the target at the first control moment includes: The speed prediction formula is used to predict the first predicted moving speed of the target at the first control moment.

6. The method according to claim 1, characterized in that The first gimbal coordinates are obtained by performing target detection on the corresponding image; The determining, based on the first gimbal coordinates corresponding to the target in each frame of the first image, respectively the real moving speed of the target corresponding to each frame of the first image, comprises: Taking the first images as current images respectively, obtaining a first coordinate difference between the first gimbal coordinates corresponding to the current image and the first gimbal coordinates corresponding to the previous first image, and obtaining a target detection time consumption of the current image; The ratio of the first coordinate difference to the target detection time is obtained as the actual moving speed of the target corresponding to the current image.

7. The method according to claim 1, characterized in that The first gimbal coordinate represents the coordinates of the preset position point of the target in the gimbal coordinate system; Before respectively determining the real moving speed of the target corresponding to each frame of the first image based on the first gimbal coordinates corresponding to the target in each frame of the first image, the method further includes: Performing target detection on the first image of each frame in sequence to obtain image coordinates of a target frame of the target in the first image of each frame; For each of the first images, convert the image coordinates of the target frame corresponding to the first image into a gimbal coordinate system to obtain a second gimbal coordinate; The second pan-tilt coordinates of the target frame are used to calculate the pan-tilt coordinates of a preset position point in the target frame as the first pan-tilt coordinates of the target corresponding to the first image.

8. The method according to any one of claims 6-7, characterized in that: The target detection is achieved by using a target detection model or a target tracking model; And / or, each frame of the first image includes a visible light image and a thermal imaging image, and the first gimbal coordinates are determined based on a target detection result of the visible light image and a target detection result of the corresponding thermal imaging image.

9. The method according to claim 1, characterized in that: The first gimbal coordinates are obtained by performing target detection on the corresponding image; After taking the first predicted moving speed as the moving speed control amount of the pan / tilt platform at the first control moment, the method further includes: Acquire a second coordinate difference between a third gimbal coordinate and a first gimbal coordinate of the target in a fourth image, wherein the fourth image is acquired by the acquisition device and is the image that has most recently completed target detection, the third gimbal coordinate represents the gimbal coordinate of the target in a fifth image, the fifth image is a frame image next to the fourth image, and the third gimbal coordinate is calculated based on the first gimbal coordinate corresponding to the fourth image and a predicted moving speed of the target corresponding to the fifth image; The ratio of the second coordinate difference to the time difference between the fourth image and the fifth image is used as the movement speed control amount of the pan / tilt head at a non-first control moment.

10. The method according to claim 1, characterized in that Before predicting a first predicted moving speed of the target at the first control moment in response to the pan / tilt platform entering a tracking state of the target, the method further includes: Receiving a control instruction sent by a radar, wherein the control instruction includes a control amount for the gimbal; After controlling the pan / tilt platform to move according to the control amount, controlling the pan / tilt platform to be stationary; Acquire a second image acquired by the acquisition device after the pan / tilt head is stationary; Acquire the first PTZ coordinates and the real moving speed corresponding to the target in the second image; and detect whether the target in the second image meets the alarm condition; In response to the target in the second image satisfying the alarm condition, it is determined that the pan / tilt head enters a tracking state for the target.

11. The method according to claim 1, characterized in that The method further comprises: While the gimbal is in the tracking state, in response to a current interruption event, the tracking state is ended, and it is monitored whether a control instruction of the radar is received, wherein the interruption event includes any one of the following: the duration of the tracking state reaches a tracking upper limit value, a new control instruction of the radar is received, a rotation instruction of the gimbal is received, and the target tracking fails.

12. A pan-tilt tracking device, characterized in that: The device comprises: An acquisition module, used for acquiring a plurality of frames of first images acquired by an acquisition device at different times, wherein each frame of the first image contains the same target, wherein the acquisition device is arranged on a pan-tilt platform; a determination module, configured to determine, based on the first gimbal coordinates corresponding to the target in each frame of the first image, the actual moving speed of the target corresponding to each frame of the first image; a prediction module, configured to predict a first predicted moving speed of the target at a first control moment in response to the gimbal entering a tracking state of the target, wherein the first predicted moving speed is predicted based on each of the real moving speeds; A control amount determination module is used to use the first predicted moving speed as the moving speed control amount of the pan / tilt platform at the first control moment.

13. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions that can be run by a processor, and the program instructions can be executed by the processor to implement the method according to any one of claims 1 to 11.

Images