Visual tracking method and apparatus for target object, and storage medium and device

Abstract

The present application belongs to the technical field of image processing. Disclosed are a visual tracking method and apparatus for a target object, and a storage medium and a device. The method comprises: extracting an image frame from a video stream captured by a gimbal camera; tracking a target object in the image frame, so as to obtain a tracking result, wherein the tracking result is used for reflecting the position of the target object in the image frame; generating an attitude parameter of the gimbal camera according to the tracking result, and adjusting the attitude of the gimbal camera according to the attitude parameter, such that a moving target object is always located at a predetermined position in a viewfinder frame of the gimbal camera, wherein the attitude parameter comprises a yaw angle parameter; and generating a motion parameter of a chassis according to the attitude parameter and the tracking result, and according to the motion parameter, controlling the chassis to move along with the target object. The present application can achieve the purpose of controlling a chassis to move along with a target object, so as to ensure that the target object is always located in the angle-of-view range of a gimbal camera, thereby continuously tracking the target object.

Description

Method, device, storage medium and equipment for visual tracking of target objects Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, storage medium, and device for visual tracking of a target object. Background Art

[0002] Visual tracking involves detecting, extracting, identifying, and tracking target objects in image sequences to obtain their motion parameters. This is then processed and analyzed to understand their behavior and complete higher-level detection tasks. Motion parameters can include position, velocity, acceleration, and trajectory.

[0003] In related technologies, cameras that capture image sequences are usually installed at fixed positions on buildings, so that we can only visually track target objects within a fixed viewing angle. If the target object leaves the viewing angle due to movement, visual tracking of the target object cannot be continued.

[0004] Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and device for visually tracking a target object, which are used to solve the problem of being unable to continue visually tracking the target object after the target object leaves the viewing range of a camera installed at a fixed position on a building. The technical solution is as follows:

[0006] In one aspect, a method for visually tracking a target object is provided, the method comprising:

[0007] Extract image frames from the video stream captured by the gimbal camera;

[0008] Tracking the target object in the image frame to obtain a tracking result;

[0009] Generating posture parameters of the pan-tilt camera according to the tracking results, and adjusting the posture of the pan-tilt camera according to the posture parameters so that the moving target object is always located at a predetermined position in the viewfinder of the pan-tilt camera;

[0010] generating motion parameters of the chassis according to the posture parameters and the tracking results, and controlling the chassis to follow the target object according to the motion parameters;

[0011] Generating motion parameters of the chassis according to the posture parameters and the tracking results, and controlling the chassis to follow the target object according to the motion parameters, includes: generating steering angle parameters of the chassis according to the posture parameters; generating speed parameters of the chassis according to the tracking results; and controlling the chassis to follow the target object according to the steering angle parameters and the speed parameters;

[0012] Generating the steering angle parameter of the chassis according to the posture parameter includes: obtaining the yaw angle parameter of the pan-tilt camera from the posture parameter; converting the yaw angle parameter into the steering angle parameter of the chassis according to a preset nonlinear function;

[0013] The converting of the yaw angle parameter into the steering angle parameter of the chassis according to a preset nonlinear function includes: the preset nonlinear function is a hyperbolic tangent function, the yaw angle range of the gimbal camera is normalized, a mapping relationship is generated according to the normalized yaw angle range and the steering angle range of the chassis, the mapping relationship representing the relationship between the normalized numerical values ​​of the steering angle and the yaw angle; an expression of a parameter z in the hyperbolic tangent function is calculated according to the yaw angle parameter; and the steering angle parameter of the chassis is generated according to the expression of the parameter z and the mapping relationship.

[0014] In a possible implementation, generating the speed parameter of the chassis according to the tracking result includes:

[0015] When the tracking result indicates that the target object is within a bounding box in the image frame, acquiring a pixel region within the bounding box from the image frame;

[0016] Calculating a percentage of a pixel area within the bounding box to a pixel area of ​​the image frame;

[0017] A speed parameter of the chassis is generated according to the percentage.

[0018] In a possible implementation, generating the speed parameter of the chassis according to the percentage includes:

[0019] Get the preset percentage threshold;

[0020] detecting whether the percentage is greater than the percentage threshold;

[0021] If the percentage is greater than the percentage threshold, generating a speed parameter with a value of zero;

[0022] If the percentage is less than or equal to the percentage threshold, a speed parameter having a value greater than zero is generated.

[0023] In a possible implementation, the method further includes:

[0024] When it is determined according to the tracking result that the target object is lost, obtaining a template image of the target object, wherein the template image is obtained by extracting an image area within a bounding box of the target object before the target object is lost;

[0025] Obtaining an image obtained by the pan-tilt camera performing a rotational scan of the current environment;

[0026] Extracting an image region having the highest similarity to the template image from the picture as the template image of the target object;

[0027] The target object is continued to be tracked according to the extracted template image.

[0028] In a possible implementation, the method further includes:

[0029] sampling a predetermined number of sample images from the plurality of image frames;

[0030] Calculating a pixel coordinate distance difference between two adjacent sampled images, wherein the pixel coordinate distance difference represents a distance difference between center points of two bounding boxes tracked at two adjacent sampling moments;

[0031] Calculating an image similarity value based on the sampled image and the template image of the target object, the image similarity value representing the similarity between image regions within two bounding boxes tracked at a sampling moment and a moment of extracting the template image;

[0032] Calculating a loss confidence of the target object based on the pixel coordinate distance difference and the image similarity value, the loss confidence being equal to a weighted sum of a loss target parameter and a similar target parameter, the loss target parameter being set based on a proportion of the comparison result by comparing the pixel coordinate distance difference with a first set threshold value, and the similar target parameter being set based on a proportion of the comparison result by comparing the image similarity value with a second set threshold value;

[0033] Whether the target object is lost is determined according to the loss confidence level.

[0034] In a possible implementation, tracking the target object in the image frame to obtain a tracking result includes:

[0035] When tracking a target object in the i-th image frame, obtaining a tracking result of the i-1-th image frame, wherein the tracking result represents a bounding box of the target object in the image frame, i≥2;

[0036] Predicting the tracking result of the i-th image frame based on the tracking result of the i-1-th image frame and the i-th image frame using a target tracking algorithm;

[0037] Update i to i+1 and continue tracking the target object in the i-th image frame.

[0038] In a possible implementation, the method further includes:

[0039] performing target detection on the first image frame, and selecting a target object to be tracked from at least one detected object;

[0040] Extracting an image region within a bounding box of the target object as a template image;

[0041] The bounding box of the target object is determined as the tracking result of the first image frame.

[0042] In one aspect, a device for visually tracking a target object is provided, the device comprising:

[0043] An image frame extraction module is used to extract image frames from the video stream captured by the PTZ camera;

[0044] a visual tracking module, configured to track a target object in the image frame to obtain a tracking result, wherein the tracking result is used to reflect a position of the target object in the image frame;

[0045] a posture adjustment module, configured to generate posture parameters of the pan-tilt camera according to the tracking results, and adjust the posture of the pan-tilt camera according to the posture parameters so that the moving target object is always located at a predetermined position in the viewfinder of the pan-tilt camera, wherein the posture parameters include a yaw angle parameter;

[0046] a chassis tracking module, configured to generate motion parameters of the chassis according to the posture parameters and the tracking results, and control the chassis to follow the target object according to the motion parameters;

[0047] The chassis tracking module is further configured to: generate a steering angle parameter of the chassis according to the posture parameter; generate a speed parameter of the chassis according to the tracking result; and control the chassis to follow the target object according to the steering angle parameter and the speed parameter;

[0048] The chassis tracking module is further configured to: obtain a yaw angle parameter of the pan / tilt camera from the posture parameter; and convert the yaw angle parameter into a steering angle parameter of the chassis according to a preset nonlinear function;

[0049] The chassis tracking module is further used to: use the preset nonlinear function as a hyperbolic tangent function to normalize the yaw angle range of the gimbal camera, generate a mapping relationship based on the normalized yaw angle range and the steering angle range of the chassis, wherein the mapping relationship represents the relationship between the normalized numerical values ​​of the steering angle and the yaw angle; calculate an expression for the parameter z in the hyperbolic tangent function based on the yaw angle parameter; and generate the steering angle parameter of the chassis based on the expression for the parameter z and the mapping relationship.

[0050] In one aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned visual tracking method of a target object.

[0051] On the one hand, a visual tracking device is provided, which includes a gimbal camera, a chassis, a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the visual tracking method of the target object as described above.

[0052] The beneficial effects of the technical solution provided by this application include at least:

[0053] When tracking a target object, the object's motion is first converted into changes in the gimbal camera's posture. This is then used to control the chassis's motion, ensuring the relative rotation angle between the chassis and the gimbal camera is zero. This ensures the chassis follows the target object's motion. This ensures the target object remains within the gimbal camera's field of view, enabling continuous tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0055] FIG1 is a schematic structural diagram of a visual tracking device;

[0056] FIG2 is a software stack framework diagram of a visual tracking device;

[0057] FIG3 is a flow chart of a method for visual tracking of a target object;

[0058] FIG4 is a flow chart of a method for visual tracking of a target object;

[0059] Figure 5 is a schematic diagram of the tanh function;

[0060] FIG6 is a structural block diagram of a device for visually tracking a target object. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0062] This embodiment discloses a visual tracking device, which includes a gimbal camera, a chassis and a controller, as shown in Figure 1. The gimbal camera can be a three-axis gimbal camera, which can form a pitch angle (pitch), a yaw angle (yaw) and a roll angle (roll) by rotating around the three axes. In this embodiment, the posture of the gimbal camera can be adjusted by adjusting the pitch angle and the yaw angle. The zoom ratio of the gimbal camera can be determined according to business needs, such as a zoom ratio of 10 times, which is not limited in this embodiment. The chassis can be any PIX chassis. The controller is a computing power platform, which can be understood as an industrial computer in a conventional autonomous driving kit. In this embodiment, NVIDIA's Jetson AGX Xavier embedded development board kit is used as the computing power platform.

[0063] The gimbal camera and chassis are respectively connected to the controller. Specifically, a bracket is provided on the chassis, and the gimbal camera is mounted on the bracket, with the front of the gimbal camera facing forward. The serial port of the gimbal camera is connected to the controller, and the serial port data of the gimbal camera is transmitted to the controller through the serial port. The HDMI (High Definition Multimedia Interface) interface of the gimbal camera is connected to the controller through an HDMI to USB (Universal Serial Bus) 3.0 acquisition card device, and the image data of the gimbal camera is transmitted to the controller through the HDMI interface. The CAN (Controller Area Network) interface of the chassis is connected to the CAN interface of the controller through a CAN line.

[0064] This embodiment discloses a software stack framework diagram for a visual tracking device, as shown in Figure 2. The software stack framework includes hardware, middleware, and software. The hardware includes a PIX chassis and a gimbal camera, the middleware includes a PIX-ROS driver and a camera ROS driver, and the software includes multiple modules. CAN data is transmitted between the PIX chassis and the PIX-ROS driver, and the yaw angle parameters, steering angle parameters, and speed parameters of the camera gimbal are transmitted between the PIX-ROS driver and the software. Serial port command data, RGB image data, and serial port data are transmitted between the gimbal camera and the camera ROS driver, and RGB image data, yaw angle parameters, steering angle parameters, and speed parameters are transmitted between the camera ROS driver and the software.

[0065] Please refer to Figure 3, which shows a method flow chart of a method for visually tracking a target object provided by an embodiment of the present application. The method for visually tracking a target object can be applied to the controller shown in Figure 1. The method for visually tracking a target object can include:

[0066] Step 301: extract image frames from a video stream captured by a pan-tilt camera.

[0067] The PTZ camera sends the video stream captured by the current environment to the controller in real time. The controller samples image frames from the video stream according to a predetermined sampling frequency and processes each image frame obtained.

[0068] Step 302: Track the target object in the image frame to obtain a tracking result.

[0069] The controller can detect a target object from one image frame and then track the target object in subsequent sampled image frames to obtain a tracking result. The tracking result is used to reflect the location of the target object in the image frame, specifically, a bounding box of the target object in the image frame.

[0070] Step 303 : generating posture parameters of the pan-tilt camera according to the tracking results, and adjusting the posture of the pan-tilt camera according to the posture parameters so that the moving target object is always located at a predetermined position in the viewfinder of the pan-tilt camera.

[0071] The attitude parameters include the yaw angle and may also include the pitch angle.

[0072] The predetermined position is a pre-set position, such as a center position, a left position, a right position, etc. In order to ensure the tracking effect, the predetermined position can be set to the center position.

[0073] Step 304 : generating motion parameters of the chassis according to the posture parameters and the tracking results, and controlling the chassis to follow the target object according to the motion parameters.

[0074] The motion parameters include steering angle and speed parameters. The steering angle is used to control the steering of the chassis, and the speed parameter is used to control the speed of the chassis.

[0075] In summary, the visual tracking method for a target object provided by the embodiments of the present application first converts the target object's motion into changes in the gimbal camera's posture. The chassis' motion is then controlled in conjunction with these changes in the gimbal camera's posture, ensuring that the relative rotation angle between the chassis and the gimbal camera is zero, thereby controlling the chassis to follow the target object's motion. This ensures that the target object remains within the gimbal camera's field of view, enabling continuous tracking of the target object.

[0076] Please refer to Figure 4, which shows a method flow chart of a method for visually tracking a target object provided by an embodiment of the present application. The method for visually tracking a target object can be applied to the controller shown in Figure 1. The method for visually tracking a target object can include:

[0077] Step 401: extract image frames from a video stream captured by a pan-tilt camera.

[0078] The PTZ camera sends the video stream captured by the current environment to the controller in real time. The controller samples image frames from the video stream according to a predetermined sampling frequency and processes each image frame obtained.

[0079] Step 402: Track the target object in the image frame to obtain a tracking result.

[0080] The controller can detect a target object from one image frame and then track the target object in subsequent sampled image frames to obtain a tracking result. The tracking result is used to reflect the location of the target object in the image frame, specifically, a bounding box of the target object in the image frame.

[0081] As shown in FIG2 , the tracking process is described by taking the controller software including the target detection module, the target tracking module, the target loss judgment module and the patrol re-detection module as an example.

[0082] After the target detection module is turned on, the target detection module performs target detection on the first image frame, selects the target object to be tracked from at least one detected object; extracts the image area within the bounding box of the target object as the template image; and determines the bounding box of the target object as the tracking result of the first image frame.

[0083] When the target detection module detects that the image frame includes an object, the object can be defaulted to the target object; when the target detection module detects that the image frame includes at least two objects, the user can be prompted to perform a selection operation and the target object can be determined based on the user's selection operation.

[0084] After determining the target object, the target detection module can send the target object's bounding box as a tracking result to the target tracking module. The tracking result includes not only the location information of the bounding box, but also the bounding box identifier (ID) to distinguish when tracking multiple target objects.

[0085] After receiving the tracking results, the target tracking module extracts the image region within the target object's bounding box from the image frame based on the tracking results as a template image. This template image can be used to determine whether the target object is lost or to retrieve it after it is lost. Details are described below and are not detailed here.

[0086] When tracking a target object in the i-th image frame, the target tracking module obtains the tracking result of the i-1th image frame. The tracking result represents the bounding box of the target object in the image frame, where i ≥ 2. The target tracking algorithm predicts the tracking result of the i-th image frame based on the tracking result of the i-1th image frame and the i-th image frame. Then, i is updated to i+1, and tracking of the target object in the i-th image frame continues. The target tracking algorithm is already very mature and will not be described in detail here.

[0087] The target tracking module sends each tracking result to the target loss judgment module. The target loss judgment module can judge whether the target object is lost based on the tracking result and the template image. The specific judgment process is as follows:

[0088] (1) A predetermined number of sample images are sampled from a plurality of image frames.

[0089] For example, if a gimbal camera shoots 30 frames per second, 180 consecutive image frames can be selected and sampled at a sampling frequency of 30 frames to obtain 6 sampled images. The number of image frames, sampling frequency, and number of sampled images can be set based on business needs and are not limited in this embodiment.

[0090] (2) Calculate the pixel coordinate distance difference based on two adjacent sampled images. The pixel coordinate distance difference represents the distance difference between the center points of the two bounding boxes tracked at two adjacent sampling moments.

[0091] The target loss detection module obtains the pixel coordinates of the center point of the bounding box of the target object in the first sampled image, obtains the pixel coordinates of the center point of the bounding box of the target object in the second sampled image, and calculates the difference between these two center point pixel coordinates to obtain a pixel coordinate distance difference value. Similarly, the target loss detection module continues to calculate the pixel coordinate distance difference values ​​corresponding to the subsequent two adjacent sampled images, ultimately obtaining multiple pixel coordinate distance differences.

[0092] The target loss judgment module compares the pixel coordinate distance difference with the first set threshold and sets the target loss parameter based on the proportion of the comparison results. Taking 6 sampling times as an example, if the 3 pixel coordinate distance differences are less than the first set threshold (for example, the empirical pixel difference is 10 pixels), the target loss parameter PixelLost = 0.5; if the 3 pixel coordinate distance differences are all greater than the first set threshold, the target loss parameter PixelLost = 1.

[0093] (3) Calculating an image similarity value based on the sampled image and the template image of the target object. The image similarity value represents the similarity between the image regions within the two bounding boxes tracked at the sampling moment and the moment when the template image is extracted.

[0094] The object loss detection module extracts the image region within the bounding box from the first sampled image and calculates the similarity between this image region and the image region in the template image, obtaining an image similarity value. Similarly, the object loss detection module continues to calculate image similarity values ​​corresponding to subsequent sampled images, ultimately obtaining multiple image similarity values.

[0095] The target loss judgment module compares the image similarity value with the second set threshold and sets the similarity target parameter based on the proportion of the comparison results. Taking 6 sampling times as an example, if the similarity values ​​of 3 images are less than the second set threshold (for example, the empirical similarity value is 17), the similarity target parameter SimilarityLow is considered to be 0.5; if the similarity values ​​of 3 images are all greater than the second set threshold, the similarity target parameter SimilarityLow is considered to be 1.

[0096] (4) Calculate the loss confidence of the target object based on the pixel coordinate distance difference and image similarity value.

[0097] The loss confidence is equal to the weighted sum of the loss target parameter and the similarity target parameter. Assuming the weight of the loss target parameter is the first coefficient a and the weight of the similarity target parameter is the second coefficient b, then the loss confidence is LossConfidence = PixelLost * a + SimilarityLow * b. a and b can be empirical values, for example, a is 0.2 and b is 0.8.

[0098] (5) Determine whether the target object is lost based on the loss confidence.

[0099] The target loss determination module determines whether the loss confidence is greater than or equal to a third set threshold. If so, the target object is determined to be lost; if less than the third set threshold, the target object is determined not to be lost. The third set threshold can be an empirical value, such as 0.5.

[0100] After determining that the target object is lost, the target loss judgment module sends a loss signal to the patrol re-detection module. The patrol re-detection module obtains a template image of the target object. The template image is obtained by extracting the image area within the bounding box of the target object before the target object is lost; obtains the picture obtained by the pan-tilt camera performing a rotational scan of the current environment; extracts the image area with the highest similarity to the template image from the picture as the template image of the target object; and continues to track the target object based on the extracted template image.

[0101] After the patrol re-detection module obtains a new template image, the tracking result corresponding to the new template image is sent to the target tracking module, and the target tracking module continues to track the target object according to the tracking result.

[0102] Step 403 : generating posture parameters of the pan-tilt camera according to the tracking results, and adjusting the posture of the pan-tilt camera according to the posture parameters so that the moving target object is always located at a predetermined position in the viewfinder of the pan-tilt camera.

[0103] The attitude parameters include the yaw angle and may also include the pitch angle.

[0104] The predetermined position is a pre-set position, such as a center position, a left position, a right position, etc. In order to ensure the tracking effect, the predetermined position can be set to the center position.

[0105] Taking the center position as an example, the controller also includes a target centering maintenance module and a gimbal camera control module. The target tracking module sends each tracking result to the target centering maintenance module. If the target centering maintenance module determines that the target object has not been lost based on the result of the target loss determination module, it adjusts the gimbal camera posture based on the tracking result.

[0106] Specifically, the target centering module obtains the pixel coordinates (x1, y1) of the cross target at the center of the current image frame, and obtains the pixel coordinates (x2, y2) of the center point of the bounding box of the target object, calculates the distance difference between the two pixel coordinates PixelsError = ((x1-x2), (y1-y2)), and obtains the pixel error values ​​in the X-axis direction and the Y-axis direction. The target centering module sends the pixel error values ​​in the X-axis direction and the Y-axis direction to the gimbal camera control module, which calculates the attitude parameters based on the pixel error values ​​in the X-axis direction and the Y-axis direction, uses the attitude parameters as the error amount for PID (proportional-integral-differential) adjustment, and performs control output in the Yaw (yaw angle) and Pitch (pitch angle) directions of the gimbal camera.

[0107] Step 404: Generate a steering angle parameter of the chassis according to the posture parameter.

[0108] The gimbal camera control module also needs to send the yaw angle parameters of the gimbal camera to the chassis control module in the controller. The chassis control module will perform linkage control with the gimbal camera control module to complete nonlinear control of the two systems to stabilize and center in continuous time.

[0109] Specifically, generating the steering angle parameter of the chassis according to the attitude parameter may include: obtaining the yaw angle parameter of the gimbal camera from the attitude parameter; and converting the yaw angle parameter into the steering angle parameter of the chassis according to a preset nonlinear function.

[0110] Converting the yaw angle parameter into a steering angle parameter of the chassis according to a preset nonlinear function may include: using the nonlinear function as a hyperbolic tangent function (tanh function), normalizing the yaw angle range of the gimbal camera, generating a mapping relationship based on the normalized yaw angle range and the steering angle range of the chassis, wherein the mapping relationship represents the relationship between the normalized numerical values ​​of the steering angle and the yaw angle; calculating an expression for a parameter z in the hyperbolic tangent function according to the yaw angle parameter; and generating the steering angle parameter of the chassis according to the expression for the parameter z and the mapping relationship.

[0111] In an example, the control range of the chassis' steering angle is [-1024, 1024], and the control range of the gimbal camera's Yaw angle is [-150, 150]. Negative values ​​in the control range represent turning left, and positive values ​​represent turning right. Because the control of the chassis and the gimbal camera are both continuous and are two nonlinear control quantities, we regard this linkage control as a continuous nonlinear control of the relative displacement between the two mechanical components. We infer the steering angle of the chassis based on the Yaw angle of the gimbal. Since the chassis is an Ackerman chassis and cannot be turned to a specified angle in place, we need to make the steering angle of the chassis follow the Yaw angle adjusted by the PID of the gimbal camera to increase nonlinearly, otherwise the steering of the chassis cannot be coordinated with the steering of the gimbal camera. Therefore, we introduce the nonlinear tanh function Its value range is [-1, 1]. As a functional relationship that associates the two control quantity ranges, the function graph is shown in Figure 5.

[0112] The chassis control module normalizes the Yaw angle control range of the gimbal camera [-150, 150] to [-1, 1]. The T1 and Yaw values ​​of the gimbal camera at the current moment are Yaw t1 , the control range of the steering angle of the chassis is [-1024, 1024], and the mapping relationship Steering t1 =Yaw t1 *1024, you can get the steering angle Steering value that the final chassis needs to control t1 Because nonlinear steering control is required, the chassis control module can t1 As the value of the left term of the tanh function, according to the tanh function, solve when Yaw t1 =g(z t1 ) t1 Then, the chassis control module will calculate the value of z t1As the final incremental amount of chassis steering, it is divided into 50 equal parts (the reason for choosing 50 equal parts is that the chassis CAN message sending frequency is 50hz, so that the chassis can receive messages of normal frequency and thus control the chassis movement), and sent to the chassis in 50 consecutive frames of messages. For example, the Z value of these 50 frames of messages is Then the corresponding chassis steering angle is And so on. And so on, the chassis control module finally sets 50 steering values ​​at a frequency of 50hz This information is sent to the chassis as steering parameters, with each frame corresponding to a steering value. This allows the chassis' steering angle and the gimbal camera's yaw angle to be controlled through a normalized mapping and the nonlinear tanh function, achieving nonlinear, incremental linkage control of the chassis' steering angle. In this loop, regardless of whether the gimbal camera follows the object to the left or right, the chassis' steering angle is linked to the gimbal camera's yaw angle through a normalized mapping and an incremental nonlinear function. The two systems will align their movements to the target object's center point when the target object is essentially stable and centered.

[0113] Step 405: Generate chassis speed parameters based on the tracking results.

[0114] Specifically, the target distance estimation module in the controller obtains the pixel area within the bounding box from the image frame; calculates the percentage of the pixel area within the bounding box to the pixel area of ​​the image frame; and generates the speed parameter of the chassis according to the percentage.

[0115] When generating the speed parameter of the chassis based on the percentage, the target distance estimation module obtains a preset percentage threshold; detects whether the percentage is greater than the percentage threshold; if the percentage is greater than the percentage threshold, generates a speed parameter with a value of zero; if the percentage is less than or equal to the percentage threshold, generates a speed parameter with a value greater than zero.

[0116] The percentage can represent the spatial distance between the target object and the gimbal camera, and the percentage threshold can represent the minimum safe distance. When the percentage is greater than the percentage threshold, the target distance estimation module determines that the spatial distance between the target object and the gimbal camera is less than the minimum safe distance and needs to control the chassis to stop moving. In other words, the value of the speed parameter is set to zero to control the chassis to slow down. When the percentage is less than or equal to the percentage threshold, the target distance estimation module determines that the spatial distance between the target object and the gimbal camera is greater than or equal to the minimum safe distance and needs to continue to control the chassis movement. In other words, the value of the speed parameter is set to greater than zero to control the chassis to continue moving.

[0117] Step 406 : Control the chassis to follow the target object according to the steering angle parameter and the speed parameter.

[0118] In the initial state, the target object, the gimbal camera, and the chassis are assumed to be basically aligned. Of course, even if the target object, the gimbal camera, and the chassis are not aligned, it will not affect the subsequent tracking process. When the target object begins to move, the gimbal camera immediately begins to rotate to track the target object. Since a nonlinear function has been created based on the nonlinear motion relationship between the gimbal camera and the chassis, the gimbal camera can be used to drive the chassis movement based on this nonlinear function, so that both the gimbal camera and the chassis tend to align with the center of the target object. In other words, from the initial position to the final position, the rotation angle between the gimbal camera and the chassis is zero, which means that there is zero rotation between the gimbal camera and the chassis.

[0119] In summary, the visual tracking method for a target object provided by the embodiments of the present application first converts the target object's motion into changes in the gimbal camera's posture. The chassis' motion is then controlled in conjunction with these changes in the gimbal camera's posture, ensuring that the relative rotation angle between the chassis and the gimbal camera is zero, thereby controlling the chassis to follow the target object's motion. This ensures that the target object remains within the gimbal camera's field of view, enabling continuous tracking of the target object.

[0120] Please refer to Figure 6, which shows a block diagram of a target object visual tracking device provided by one embodiment of the present application. The target object visual tracking device can be applied to the controller shown in Figure 1. The target object visual tracking device can include:

[0121] An image frame extraction module 610 is used to extract image frames from a video stream captured by a PTZ camera;

[0122] A visual tracking module 620 is configured to track a target object in an image frame and obtain a tracking result, wherein the tracking result is used to reflect the position of the target object in the image frame;

[0123] a posture adjustment module 630 for generating posture parameters of the PTZ camera based on the tracking results, and adjusting the posture of the PTZ camera based on the posture parameters so that the moving target object is always located at a predetermined position in the viewfinder of the PTZ camera, wherein the posture parameters include a yaw angle parameter;

[0124] The chassis tracking module 640 is used to generate motion parameters of the chassis according to the posture parameters and the tracking results, and control the chassis to follow the target object according to the motion parameters.

[0125] In an optional embodiment, the chassis tracking module 640 is further configured to:

[0126] Generate chassis steering angle parameters based on posture parameters;

[0127] Generate chassis speed parameters according to the tracking results;

[0128] The chassis is controlled to follow the target object according to the steering angle parameters and speed parameters.

[0129] In an optional embodiment, the chassis tracking module 640 is further configured to:

[0130] Get the yaw angle parameter of the gimbal camera from the attitude parameter;

[0131] The yaw angle parameter is converted into the steering angle parameter of the chassis according to a preset nonlinear function.

[0132] In an optional embodiment, the chassis tracking module 640 is further configured to:

[0133] The nonlinear function is a hyperbolic tangent function, which normalizes the yaw angle range of the gimbal camera. A mapping relationship is generated based on the normalized yaw angle range and the steering angle range of the chassis. The mapping relationship represents the relationship between the normalized values ​​of the steering angle and the yaw angle.

[0134] Calculate the expression of the parameter z in the hyperbolic tangent function according to the yaw angle parameter;

[0135] The steering angle parameter of the chassis is generated according to the expression and mapping relationship of the parameter z.

[0136] In an optional embodiment, the chassis tracking module 640 is further configured to:

[0137] When the tracking result indicates that the target object is within the bounding box in the image frame, a pixel region within the bounding box is obtained from the image frame;

[0138] Calculate the percentage of the pixel area within the bounding box to the pixel area of ​​the image frame;

[0139] Generates chassis speed parameters based on percentages.

[0140] In an optional embodiment, the chassis tracking module 640 is further configured to:

[0141] Get the preset percentage threshold;

[0142] Check whether the percentage is greater than the percentage threshold;

[0143] If the percentage is greater than the percentage threshold, a speed parameter with a value of zero is generated;

[0144] If the percentage is less than or equal to the percentage threshold, a velocity parameter with a value greater than zero is generated.

[0145] In an optional embodiment, the visual tracking module 620 is further configured to:

[0146] When it is determined that the target object is lost according to the tracking result, a template image of the target object is obtained, where the template image is obtained by extracting an image area within a bounding box of the target object before the target object is lost;

[0147] Get the image obtained by the pan-tilt camera rotating and scanning the current environment;

[0148] Extracting the image area with the highest similarity to the template image from the picture as the template image of the target object;

[0149] Continue tracking the target object based on the extracted template image.

[0150] In an optional embodiment, the visual tracking module 620 is further configured to:

[0151] sampling a predetermined number of sample images from the plurality of image frames;

[0152] Calculate the pixel coordinate distance difference based on two adjacent sampled images. The pixel coordinate distance difference represents the distance difference between the center points of the two bounding boxes tracked at two adjacent sampling moments.

[0153] Calculate an image similarity value based on the sampled image and the template image of the target object. The image similarity value represents the similarity between the image regions within the two bounding boxes tracked at the sampling moment and the moment when the template image is extracted.

[0154] Calculate the loss confidence of the target object based on the pixel coordinate distance difference and the image similarity value. The loss confidence is equal to the weighted sum of the loss target parameter and the similarity target parameter. The loss target parameter is set based on the proportion of the comparison results by comparing the pixel coordinate distance difference with a first set threshold. The similarity target parameter is set based on the proportion of the comparison results by comparing the image similarity value with a second set threshold.

[0155] Determine whether the target object is lost based on the loss confidence.

[0156] In an optional embodiment, the visual tracking module 620 is further configured to:

[0157] When tracking the target object in the i-th image frame, obtain the tracking result of the i-1-th image frame. The tracking result represents the bounding box of the target object in the image frame, i≥2;

[0158] Use the target tracking algorithm to predict the tracking result of the i-th image frame based on the tracking result of the i-1th image frame and the i-th image frame;

[0159] Update i to i+1 and continue tracking the target object in the i-th image frame.

[0160] In an optional embodiment, the visual tracking module 620 is further configured to:

[0161] performing target detection on the first image frame, and selecting a target object to be tracked from at least one detected object;

[0162] Extract the image area within the bounding box of the target object as the template image;

[0163] The bounding box of the target object is determined as the tracking result of the first image frame.

[0164] In summary, the visual tracking device for a target object provided by the embodiments of the present application, during the tracking process, first converts the target object's motion into changes in the gimbal camera's posture. It then controls the chassis's motion in conjunction with these changes in the gimbal camera's posture, ensuring that the relative rotation angle between the chassis and the gimbal camera is zero, thereby achieving the goal of controlling the chassis to follow the target object's motion. This ensures that the target object remains within the gimbal camera's field of view, enabling continuous tracking of the target object.

[0165] One embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned visual tracking method of a target object.

[0166] One embodiment of the present application provides a visual tracking device, which includes a gimbal camera, a chassis, a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the visual tracking method of the target object as described above.

[0167] It should be noted that the above embodiments provide a visual tracking device for a target object, and only illustrate the division of the above functional modules when performing visual tracking of the target object. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the visual tracking device for a target object can be divided into different functional modules to complete all or part of the functions described above. In addition, the visual tracking device for a target object provided in the above embodiments and the embodiment of the visual tracking method for a target object are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0168] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0169] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the embodiments of the present application.

Claims

1. A method for visual tracking of a target object, characterized in that: The method comprises: Extract image frames from the video stream captured by the gimbal camera; Tracking the target object in the image frame to obtain a tracking result, wherein the tracking result is used to reflect the position of the target object in the image frame; Generating attitude parameters of the pan-tilt camera according to the tracking results, and adjusting the attitude of the pan-tilt camera according to the attitude parameters so that the moving target object is always located at a predetermined position in the viewfinder of the pan-tilt camera, wherein the attitude parameters include yaw angle parameters; Generate motion parameters of the chassis according to the posture parameters and the tracking results, and control the chassis to follow the target object according to the motion parameters; Generating motion parameters of the chassis according to the posture parameters and the tracking results, and controlling the chassis to follow the target object according to the motion parameters, includes: generating steering angle parameters of the chassis according to the posture parameters; generating speed parameters of the chassis according to the tracking results; and controlling the chassis to follow the target object according to the steering angle parameters and the speed parameters; Generating the steering angle parameter of the chassis according to the attitude parameter includes: acquiring the yaw angle parameter of the gimbal camera from the attitude parameter; converting the yaw angle parameter into the steering angle parameter of the chassis according to a preset nonlinear function; The method of converting the yaw angle parameter into the steering angle parameter of the chassis according to a preset nonlinear function includes: the preset nonlinear function is a hyperbolic tangent function, the yaw angle range of the gimbal camera is normalized, a mapping relationship is generated according to the normalized yaw angle range and the steering angle range of the chassis, the mapping relationship represents the relationship between the normalized numerical values ​​of the steering angle and the yaw angle; an expression of a parameter z in the hyperbolic tangent function is calculated according to the yaw angle parameter; and the steering angle parameter of the chassis is generated according to the expression of the parameter z and the mapping relationship.

2. The method for visual tracking of a target object according to claim 1, characterized in that: Generating the speed parameter of the chassis according to the tracking result includes: When the tracking result indicates that the target object is within a bounding box in the image frame, acquiring a pixel region within the bounding box from the image frame; Calculating a percentage of a pixel area within the bounding box to a pixel area of ​​the image frame; A speed parameter of the chassis is generated according to the percentage.

3. The method for visual tracking of a target object according to claim 2, characterized in that: Generating the speed parameter of the chassis according to the percentage includes: Get the preset percentage threshold; detecting whether the percentage is greater than the percentage threshold; If the percentage is greater than the percentage threshold, generating a speed parameter with a value of zero; If the percentage is less than or equal to the percentage threshold, a speed parameter having a value greater than zero is generated.

4. The method for visual tracking of a target object according to claim 1, characterized in that: The method further comprises: When it is determined according to the tracking result that the target object is lost, acquiring a template image of the target object, wherein the template image is obtained by extracting an image area within a boundary box of the target object before the target object is lost; Obtaining a picture obtained by the pan-tilt camera performing a rotational scan of the current environment; Extracting an image region having the highest similarity to the template image from the picture as the template image of the target object; The target object continues to be tracked according to the extracted template image.

5. The method for visual tracking of a target object according to claim 4, characterized in that: The method further comprises: sampling a predetermined number of sample images from a plurality of image frames; Calculating a pixel coordinate distance difference value based on two adjacent sampled images, wherein the pixel coordinate distance difference value represents a distance difference value between center points of two bounding boxes tracked at two adjacent sampling moments; Calculating an image similarity value based on the sampled image and the template image of the target object, the image similarity value representing the similarity between image regions within two bounding boxes tracked at a sampling moment and a moment of extracting the template image; The loss confidence of the target object is calculated according to the pixel coordinate distance difference and the image similarity value, wherein the loss confidence is equal to the weighted sum of the lost target parameter and the similar target parameter, and the lost target parameter is obtained by comparing the pixel coordinate distance difference with the first set threshold value, and according to the comparison result The similarity target parameter is set by comparing the image similarity value with the second set threshold value and according to the proportion of the comparison result; Whether the target object is lost is determined according to the loss confidence.

6. The method for visual tracking of a target object according to any one of claims 1 to 5, characterized in that: Tracking the target object in the image frame to obtain a tracking result includes: When tracking the target object in the i-th image frame, obtaining the tracking result of the i-1-th image frame, the tracking result represents the bounding box of the target object in the image frame, i≥2; Predicting the tracking result of the i-th image frame based on the tracking result of the i-1-th image frame and the i-th image frame using a target tracking algorithm; Update i to i+1 and continue tracking the target object in the i-th image frame.

7. The method for visual tracking of a target object according to claim 6, characterized in that: The method further comprises: Performing target detection on the first image frame, and selecting a target object to be tracked from at least one detected object; Extracting an image region within a bounding box of the target object as a template image; A bounding box of the target object is determined as a tracking result of the first image frame.

8. A visual tracking device for a target object, characterized in that: The device comprises: An image frame extraction module is used to extract image frames from the video stream captured by the PTZ camera; A visual tracking module, used to track the target object in the image frame to obtain a tracking result, wherein the tracking result is used to reflect the position of the target object in the image frame; A posture adjustment module, used for generating posture parameters of the pan-tilt camera according to the tracking result, and adjusting the posture of the pan-tilt camera according to the posture parameters so that the moving target object is always located at a predetermined position in the viewfinder of the pan-tilt camera, wherein the posture parameters include yaw angle parameters; A chassis tracking module, used to generate a motion parameter of the chassis according to the posture parameter and the tracking result, and control the chassis to follow the target object according to the motion parameter; The chassis tracking module is further used to: generate a steering angle parameter of the chassis according to the posture parameter; generate a speed parameter of the chassis according to the tracking result; and generate a speed parameter of the chassis according to the steering angle parameter and the The speed parameter controls the chassis to follow the movement of the target object; The chassis tracking module is further used to: obtain a yaw angle parameter of the pan / tilt camera from the posture parameter; and convert the yaw angle parameter into a steering angle parameter of the chassis according to a preset nonlinear function; The chassis tracking module is also used for: the preset nonlinear function is a hyperbolic tangent function, the yaw angle range of the gimbal camera is normalized, a mapping relationship is generated according to the normalized yaw angle range and the steering angle range of the chassis, the mapping relationship represents the relationship between the normalized numerical values ​​of the steering angle and the yaw angle; the expression of the parameter z in the hyperbolic tangent function is calculated according to the yaw angle parameter; the steering angle parameter of the chassis is generated according to the expression of the parameter z and the mapping relationship.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the visual tracking method of the target object as described in any one of claims 1 to 7.

10. A visual tracking device, characterized in that: The visual tracking device includes a gimbal camera, a chassis, a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the visual tracking method of the target object as described in any one of claims 1 to 7.

Images