A pan-tilt target tracking method based on off-target amount

Abstract

This invention discloses a gimbal target tracking method based on miss distance. In normal tracking mode, a two-dimensional parameter level table is constructed by calculating the miss distance between the target bounding box center and the image center and its inter-frame change rate, indexed by the absolute value of the miss distance and the absolute value of the change rate. The PID control parameters adapted to the target motion state are determined in real-time based on the current miss distance dataset, and the parameters of adjacent control cycles are smoothly transitioned using an exponentially weighted moving average method before generating gimbal rotation commands. When the target is not detected for several consecutive frames, a loss handling mode is entered. The current frame position is predicted based on historical trajectories, and a virtual miss distance is generated to maintain predictive tracking. Simultaneously, re-acquisition is determined within the expanded search area with a higher confidence threshold, and the control parameters are smoothly restored upon successful re-acquisition.

Description

Technical Field

[0001] This invention belongs to the field of visual servo control technology and relates to a gimbal target tracking method based on miss distance. Background Technology

[0002] In applications such as intelligent security and drone tracking, gimbals need to adjust their attitude in real time based on visual information to keep the target centered in the field of view. The most commonly used technical solution is visual positioning, position command, and gimbal servoing paradigm: the host detects the target through visual algorithms and calculates its position coordinates in the image, converts the coordinates into a gimbal rotation command and sends it to the actuator to drive the motor to the designated position; at the same time, the gimbal's built-in inertial measurement unit can maintain its attitude on its own when no command is received, achieving basic stabilization.

[0003] First, there is a significant delay throughout the entire chain from image acquisition, target detection, position calculation to command issuance. By the time a command is generated, the target has already moved to a new position, causing the gimbal to constantly chase the target, with particularly severe deviations during high-speed target maneuvers. Second, to accommodate different motion states, servo systems typically use fixed PID control parameters. However, these parameters cannot simultaneously meet the requirements of rapid convergence under large deviations and zero overshoot under small deviations. Increasing the proportional coefficient can speed up the response, but it can cause overshoot and oscillations when the target's motion changes abruptly, reducing image quality and shortening motor life. Furthermore, when the target is briefly occluded, existing solutions usually stop the action directly or maintain the last command, lacking a prediction of the target's motion trend and an active search mechanism, resulting in a high probability of permanent target loss.

[0004] Therefore, there is an urgent need for a gimbal target tracking solution that can adaptively adjust control characteristics according to the real-time motion state of the target and has the ability to predict and reacquire the target after a brief loss, in order to solve the shortcomings of the current solution in terms of dynamic accuracy, control stability and tracking continuity. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention proposes a gimbal target tracking method based on miss distance.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a gimbal target tracking method based on miss distance, comprising the following steps:

[0007] Capture real-time images output by the camera mounted on the pan-tilt unit;

[0008] Obtain the target bounding box output by the target detection model based on the real-time image;

[0009] Based on the center coordinates of the target bounding box and the center coordinates of the real-time image, calculate the horizontal and vertical miss distances, and calculate the inter-frame variation rate of the horizontal and vertical miss distances;

[0010] Based on the horizontal miss amount, the vertical miss amount, the inter-frame change rate of the horizontal miss amount, and the inter-frame change rate of the vertical miss amount, generate the miss amount dataset for the current frame.

[0011] Based on the miss distance dataset, determine a set of PID control parameters that are adapted to the current motion state;

[0012] Based on the miss distance dataset and the set of PID control parameters, a gimbal rotation command is generated;

[0013] The gimbal rotation command is sent to the gimbal drive module to drive the gimbal to rotate so that the target returns to the center of the real-time image.

[0014] Specifically, determining a set of PID control parameters adapted to the current motion state based on the miss distance dataset includes:

[0015] A two-dimensional parameter level table is constructed by jointly indexing the absolute value of the miss distance and the absolute value of the rate of change. The two-dimensional parameter level table stores multiple control regions and a set of preset PID control parameters corresponding to each control region.

[0016] Based on the absolute value of the horizontal miss amount and the absolute value of the inter-frame change rate of the horizontal miss amount in the miss amount data, the first target PID control parameter acting in the horizontal direction is determined by looking up the table in the two-dimensional parameter range table.

[0017] Based on the absolute value of the vertical miss distance and the absolute value of the inter-frame change rate of the vertical miss distance in the miss distance dataset, the second target PID control parameters acting in the vertical direction are determined by looking up the table in the two-dimensional parameter range table.

[0018] Specifically, determining a set of PID control parameters adapted to the current motion state based on the miss distance dataset further includes:

[0019] Obtain the first and second historical PID control parameters that were the final action of the previous control cycle;

[0020] Based on the first target PID control parameters and the first historical PID control parameters, an exponential weighted moving average method is used for smoothing to obtain a set of PID control parameters that are finally applied to the horizontal direction control.

[0021] Based on the second target PID control parameters and the second historical PID control parameters, an exponentially weighted moving average method is used for smoothing to obtain a set of PID control parameters that ultimately act on the vertical direction control.

[0022] Specifically, the multiple control areas in the two-dimensional parameter range table include a fine-tuning area, a stable area, and a high dynamic area;

[0023] The preset PID control parameters corresponding to the fine-tuning zone satisfy the following: the proportional coefficient is less than the first proportional threshold, the integral coefficient is less than the first integral threshold, and the derivative coefficient is less than the first derivative threshold.

[0024] The preset PID control parameters corresponding to the stable region satisfy the following: the proportional coefficient belongs to the second proportional range, the integral coefficient belongs to the second integral range, and the derivative coefficient belongs to the second derivative range.

[0025] The preset PID control parameters corresponding to the high dynamic range satisfy the following: the proportional coefficient belongs to the third proportional range, the integral coefficient belongs to the third integral range, and the derivative coefficient belongs to the third derivative range.

[0026] Wherein, the lower limit of the third proportional range is greater than the upper limit of the second proportional range, and the lower limit of the third differential range is greater than the upper limit of the second differential range.

[0027] Specifically, a gimbal target tracking method based on miss distance also includes:

[0028] When the target detection model does not output a bounding box of the same target identifier with a confidence level higher than the first confidence threshold in consecutive image frames of the first frame number threshold, it determines that the target is lost and enters the target loss processing mode.

[0029] In the target loss handling mode, the proportional coefficient in the current set of PID control parameters is reduced to a first preset percentage in the normal tracking mode, the derivative coefficient is reduced to a second preset percentage in the normal tracking mode, and the integral coefficient is set to zero.

[0030] In the target loss processing mode, the predicted position of the current frame is obtained based on the historical trajectory sequence before the target is lost. The virtual miss distance is calculated based on the predicted position of the current frame and the center coordinates of the real-time image. The virtual miss distance is used as the miss distance dataset to generate the gimbal rotation command.

[0031] Specifically, obtaining the current frame prediction position based on the historical trajectory sequence before the target was lost includes:

[0032] Obtain the historical target center point coordinate sequence before the target was lost, based on the last preset second frame number threshold.

[0033] The historical target center point coordinate sequence is input into a Kalman filter to obtain the state estimation vector output by the Kalman filter.

[0034] The predicted coordinates of the target in the horizontal direction and the predicted coordinates in the vertical direction are extracted from the first preset element position and the second preset element position of the state estimation vector, respectively, and used as the predicted position of the current frame.

[0035] Specifically, a gimbal target tracking method based on miss distance also includes:

[0036] In the target loss processing mode, the expanded search area is determined based on the predicted position of the current frame, and the size of the expanded search area is a first preset multiple of the target bounding box size of the last frame before the target was lost;

[0037] The expanded search area is used as the input area for the target detection model to perform target detection.

[0038] Specifically, a gimbal target tracking method based on miss distance also includes:

[0039] When the target detection model detects a bounding box of the same target identifier with a confidence level higher than the second confidence threshold again within the expanded search area under the target loss processing mode, it determines that the recapture is successful.

[0040] The attenuated proportional coefficient and differential coefficient are smoothly restored to their values ​​under the normal tracking mode, and the integral coefficient is restored to a non-zero value, thus exiting the target loss processing mode;

[0041] When the duration of the target loss processing mode exceeds a first duration threshold and recapture is still unsuccessful, the generation of the gimbal rotation command is stopped, and the gimbal is controlled to enter the scanning search state.

[0042] Specifically, the different steps of the method are executed in parallel by multiple independent threads deployed on the same embedded platform, wherein:

[0043] The first thread performs the real-time image acquisition operation at a first frequency;

[0044] The second thread performs target detection and tracking operations and outputs the target bounding box.

[0045] The third thread executes the operations of generating the miss distance dataset, determining the set of PID control parameters, and generating the gimbal rotation command at a second frequency higher than the first frequency.

[0046] The second thread and the third thread interact with each other through a data buffer with a preset capacity. The third thread reads the latest target information from the data buffer through atomic operations. When the atomic operation fails to read the target information, the historical target information successfully read in the previous cycle is used to perform subsequent operations.

[0047] Specifically, the third thread and the fourth thread used to send the gimbal rotation command are configured with a first scheduling strategy and a first priority;

[0048] The first thread and the second thread are configured with the second scheduling policy and the second priority;

[0049] The value of the first priority is higher than the value of the second priority.

[0050] Compared with the prior art, the present invention has the following beneficial effects: by constructing a two-dimensional parameter level table jointly indexed by the absolute value dimension of the miss amount and the absolute value dimension of the rate of change, and by determining the appropriate PID control parameters in real time based on the current miss amount dataset, the adaptive adjustment of control parameters with the target motion state is realized, taking into account both fast convergence when the deviation is large and overshoot-free control when the deviation is small.

[0051] By employing the exponentially weighted moving average method to smoothly transition the PID control parameters between adjacent control cycles, parameter abrupt changes during control region switching are avoided, effectively suppressing overshoot and mechanical oscillation of the gimbal servo system.

[0052] By predicting the current frame position based on historical trajectory and generating a virtual miss distance when the target is lost, the predictive tracking of the gimbal is maintained. At the same time, re-acquisition is determined with a higher confidence threshold in the expanded search area. Compared with the existing technology that directly stops or performs erroneous actions after the target is lost, the success rate and real-time performance of re-acquisition after occlusion are improved. Attached Figure Description

[0053] Figure 1 This is a block diagram of the gimbal target tracking method based on miss distance of the present invention;

[0054] Figure 2 This is the logic diagram of the two-dimensional state space partitioning and adaptive parameter selection of the present invention;

[0055] Figure 3 This is a schematic diagram of the multi-threaded asynchronous pipeline architecture and timing coordination of the present invention;

[0056] Figure 4 This is a flowchart of the dual-mode switching logic for target loss handling in this invention;

[0057] Figure 5 This is a schematic diagram of the data flow of the Kalman filter predictor of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] like Figures 1-5 As shown, the technical solution adopted in this invention is as follows: A gimbal target tracking method based on miss distance, comprising the following steps:

[0060] S1: Collects real-time images output by the camera mounted on the PTZ.

[0061] During system operation, raw video streams are acquired from a camera mounted on the gimbal at a fixed frequency of 30 frames per second (30fps). The video stream has an image resolution of 1920 pixels × 1080 pixels and uses the YUYV color space. The camera, as the system's visual perception front-end, forms a data transmission and reception interface with the subsequent target detection and tracking threads. Specifically, the real-time image data stream output from this step serves as input data for the target detection and tracking modules in subsequent steps. The continuous and stable acquisition of these real-time images is the data foundation and prerequisite for the entire closed-loop control loop to perform target tracking based on visual feedback, ensuring that subsequent operations such as miss distance calculation, target tracking status maintenance, and loss handling all have usable raw image frames.

[0062] S2: Obtain the target bounding box output by the target detection model based on the real-time image.

[0063] The system identifies targets to be tracked from real-time images and outputs bounding boxes describing the spatial location of the target within the image. It integrates two sub-modules: a target detection model and a target tracker. The target detection model performs visual reasoning on a single frame of image to identify potential target instances; the target tracker associates detection results with the same identity across consecutive video frames to maintain consistency in target identifiers.

[0064] In one implementation, the target detection model uses the lightweight convolutional neural network model YOLOv8n and leverages the TensorRT engine for inference acceleration to adapt to the computing resources of the embedded platform NVIDIA Jetson Orin Nano and meet real-time requirements. The model's processing flow includes: adjusting the size of the single-frame real-time image output from step one to 640 pixels × 640 pixels before using it as model input; further post-processing the original detection results output by the model using a non-maximum suppression algorithm, where the suppression threshold—Intersection over Union (IoU) threshold—is set to 0.45 to remove redundant detection boxes for the same target, and the confidence threshold is set to 0.7 to filter out erroneous detection results with low confidence. The output after the above processing is the preliminary target bounding box given by the target detection model in that frame image.

[0065] The target tracker used is the ByteTrack tracker, whose key hyperparameter, the trajectory buffer, is set to 30 frames. This means that if a target briefly disappears, its trajectory identifier can be retained in the buffer for 30 frames. If the target is detected again during this period, its original identifier can be recovered, thus achieving a stable connection of the target identity. In a continuous video stream, the tracker performs association matching between the detection results of the current frame and the maintained trajectory, and outputs the final target bounding box with a unique target identifier.

[0066] The target bounding box output in this step is quantitatively described by four coordinate values: the x and y coordinates of the upper left corner of the bounding box, and the x and y coordinates of the lower right corner of the bounding box. This set of coordinate data serves as the direct input parameters for subsequent miss distance calculation steps and is a fundamental component in building a closed-loop control system based on miss distance feedback. The real-time performance of its detection and the accuracy of its positioning directly affect the performance of the entire tracking system.

[0067] S3: Calculate the horizontal and vertical miss distances based on the center coordinates of the target bounding box and the center coordinates of the real-time image, and calculate the inter-frame variation rate of the horizontal and vertical miss distances.

[0068] First, determine the coordinates of the target center and the image center. The target center coordinates are calculated from the coordinates of the target bounding box, specifically by adjusting the x-coordinate of the top-left corner of the bounding box. and the bottom right x-coordinate Add them together and divide by two to get the x-coordinate of the target center. ; Set the y-coordinate of the top left corner of the bounding box and the bottom right ordinate Add them together and divide by two to get the ordinate of the target center. The coordinates of the image center are determined by the size of the real-time image; for an image with a width of... pixels, height is The image of pixels, its image center x-coordinate equal Divide by two, the ordinate of the image center equal Divide by two.

[0069] Next, calculate the miss distance in the horizontal and vertical directions. Set the x-coordinate of the target center... Subtract the x-coordinate of the image center The difference obtained is defined as the horizontal miss distance. The unit is pixels; the vertical coordinate of the target center is... Subtract the ordinate of the image center The difference obtained is defined as the vertical miss distance. The unit is pixels. Horizontal miss distance. The numerical value represents the pixel distance of the target from the center of the field of view in the horizontal direction, and its sign indicates the direction of deviation; the miss distance in the vertical direction... The numerical value represents the pixel distance by which the target deviates from the center of the field of view in the vertical direction, and its sign indicates the direction of deviation. The magnitude of the miss distance directly reflects the degree to which the target deviates from the image center and serves as the error basis for subsequent generation of gimbal control commands to correct the deviation.

[0070] Finally, calculate the inter-frame rate of change of the miss distance. Let the horizontal miss distance of the previous frame be... The vertical miss distance in the previous frame was The time interval between the current frame and the previous frame is The horizontal miss distance of the current frame. Subtract the horizontal miss distance from the previous frame The difference obtained is divided by the time interval. The inter-frame variation rate of the horizontal miss distance was obtained. ; The vertical miss distance of the current frame Subtract the vertical miss distance from the previous frame The difference obtained is divided by the time interval. The inter-frame variation rate of the vertical miss distance was obtained. With a fixed frame rate of 30fps, one frame corresponds to a time interval of 1 / 30th of a second. Therefore, the rate of change expressed in pixels per frame can be converted to pixels per second. Values ​​involving rate of change thresholds are all in pixels per second; to convert to pixels per frame, divide the value in pixels per second by 30. Inter-frame rate of change. and It reflects the target's speed and directional trend relative to the image center, providing crucial dynamic information for subsequent steps to dynamically adjust control parameters based on the motion state.

[0071] To facilitate consistent terminology in subsequent steps, when the horizontal and vertical directions are not distinguished and only the general properties of the miss distance are discussed, the symbol will be used. Generally refers to the amount of missed target, expressed in symbols. Generally refers to the rate of change of the miss distance. This is used when determining control parameters for the horizontal and vertical directions separately. and These refer to the miss distance and its rate of change in the corresponding directions, i.e., for the horizontal direction... = , = For the vertical direction = , = .

[0072] The spatial location information provided by the target bounding box output by the visual perception module is quantized and converted into the miss distance. and the rate of change of the miss amount This pair of error state signals constitutes complete feedback information. It serves as the direct input basis for subsequent adaptive control law to perform two-dimensional state space partitioning and dynamic adjustment of PID control parameters, laying the foundation for realizing closed-loop target tracking control based on miss feedback.

[0073] S4: Generate the miss amount dataset for the current frame based on the horizontal miss amount, the vertical miss amount, the inter-frame change rate of the horizontal miss amount, and the inter-frame change rate of the vertical miss amount.

[0074] The input data comes from calculations performed previously, specifically including four quantified error state variables: horizontal miss distance. Vertical miss distance Inter-frame variation rate of horizontal miss distance and the inter-frame variation rate of vertical miss distance Among them, the horizontal miss distance and vertical miss distance These represent the pixel distances of the target center from the center of the real-time image in the horizontal and vertical directions, respectively, in pixels; the inter-frame rate of change of the horizontal miss distance. Inter-frame variation rate of vertical miss distance These represent the rate of change of the aforementioned miss distance between two consecutive frames.

[0075] The processing steps in this step involve aggregating the four error state variables calculated in the current frame into a single data set, denoted as the miss distance dataset. This miss distance dataset serves as the direct input for determining a set of PID control parameters adapted to the current motion state in subsequent steps. Subsequent steps will use the miss distance data from this dataset... Determine the magnitude of the current deviation based on the rate of change of the miss distance. The process involves determining the current trend of deviation to identify the target's current motion state region in a two-dimensional state space, and then dynamically adjusting the PID control parameters accordingly. The technical contribution of this step lies in providing visual feedback information in a structured and standardized data format to the control law calculation stage. This allows for the joint examination of error magnitude and trend, providing complete feedback observation input for the subsequent implementation of an adaptive parameter adjustment mechanism based on a two-dimensional state space.

[0076] S5: Based on the miss distance dataset, determine a set of PID control parameters that are suitable for the current motion state.

[0077] The off-target dataset for the current frame contains off-target data. and the rate of change of the miss amount The output is a set of PID control parameters for the current control cycle, including the proportional gain. Integral coefficient and differential coefficients These parameters will be directly used in the calculation of the gimbal rotation speed command in subsequent steps.

[0078] Off-target amount and the rate of change of the miss amount Two observable quantities reflecting the target's motion state: miss distance and distance from the target. This represents the current distance of the target from the image center; a larger value indicates a greater deviation, requiring stronger control to correct it quickly. (Missing target change rate) The values ​​represent the target's velocity and directional trend relative to the image center. Higher values ​​indicate greater target maneuverability, requiring stronger damping to suppress overshoot. By jointly examining these two observations, it's possible to identify whether the target is currently in a state of slight swaying, steady motion, or high-speed maneuvering, and then select PID control parameters suitable for that state.

[0079] The step of determining a set of PID control parameters adapted to the current motion state based on the miss distance dataset specifically includes:

[0080] A two-dimensional parameter level table is constructed, which is jointly indexed by the absolute value of the miss distance and the absolute value of the rate of change. The two-dimensional parameter level table stores multiple control regions and a set of preset PID control parameters corresponding to each control region.

[0081] A two-dimensional parameter level table is established. This table is a pre-defined and stored lookup table data structure used to quickly retrieve the corresponding control parameters based on the quantitative indicators of the current motion state.

[0082] This two-dimensional parameter level table is composed of two index dimensions: the first dimension is the absolute value of the miss distance, and its index value is the miss distance in the horizontal direction. The absolute value or vertical miss distance The first dimension is the absolute value of the deviation, expressed in pixels, representing the magnitude of the deviation; the second dimension is the absolute value of the rate of change, whose index value is the inter-frame rate of change of the horizontal miss distance. The absolute value or the rate of change of the vertical miss distance between frames. The absolute value, measured in pixels per second, represents the rate at which the deviation changes.

[0083] The two-dimensional parameter range table stores multiple pre-divided control regions. Each control region occupies a defined range in a two-dimensional state space consisting of the absolute value of the miss distance and the absolute value of the rate of change. Each control region corresponds to a set of preset PID control parameters, including preset proportional coefficients. Preset integral coefficients and preset differential coefficients .

[0084] The two-dimensional parameter setting table internally stores three control areas: the fine-tuning area, the stable area, and the high dynamic range area. These three control areas correspond to three different motion states of the target in the field of view. The fine-tuning area corresponds to a quasi-static state where the target center has basically converged to the vicinity of the image center, with only minor deviations and slow changes; the stable area corresponds to a normal tracking state where the target moves at a moderate speed or with moderate deviations; and the high dynamic range area corresponds to a violent motion state where the target exhibits large deviations or high-speed maneuvers.

[0085] For the fine-tuning range, the corresponding set of preset PID control parameters satisfies the following numerical relationship: proportional coefficient The value of is less than the first proportional threshold, and the integral coefficient is... The value of is less than the first integration threshold, and the differential coefficient The value of is less than the first differential threshold. This set of restrictions makes the control parameters in the fine-tuning region exhibit low gain characteristics. Under this parameter configuration, the controller responds more gently to small deviations, avoiding noise amplification and mechanical oscillation caused by excessive gain. It is mainly used to eliminate steady-state errors and maintain the stability of the target in the center region of the image.

[0086] For the stable region, the corresponding set of preset PID control parameters satisfies the following numerical relationship: proportional coefficient The value of falls within the second proportional range, and the integral coefficient The value of falls within the second integral range, and the differential coefficient The value of falls within the second differential range. The upper limit of the second proportional range is greater than the first proportional threshold, and the lower limit of the second proportional range is greater than the first proportional threshold; the upper limit of the second integral range is greater than the first integral threshold, and the lower limit of the second integral range is greater than the first integral threshold; the upper limit of the second differential range is greater than the first differential threshold, and the lower limit of the second differential range is greater than the first differential threshold. The parameters in the steady-state region are generally higher than those in the fine-tuning region, exhibiting moderate gain characteristics, achieving a balance between response speed and stability, and are suitable for tracking scenarios involving normal target motion.

[0087] For the high dynamic range, the corresponding set of preset PID control parameters satisfies the following numerical relationship: proportional coefficient The value of falls within the third proportional range, and the integral coefficient The value of falls within the third integral range, and the differential coefficient The value of falls within the third derivative range. The core constraint of the high dynamic range parameters is that the lower limit of the third proportional range is greater than the upper limit of the second proportional range, and the lower limit of the third derivative range is greater than the upper limit of the second derivative range. This constraint ensures the proportional coefficient in the high dynamic range. and differential coefficients All parameters are generally higher than those in the steady-state region, exhibiting high proportionality and high differential characteristics. Under this parameter configuration, the proportionality coefficient is relatively high. Provides a powerful correction drive to achieve rapid convergence of large deviations, with a high differential coefficient. It provides strong damping to suppress overshoot and oscillations caused by increased proportional gain. Simultaneously, the integral coefficient in the high dynamic range... If it falls into the third integration range, its value can be higher than the integration coefficient in the steady region, so as to accelerate the integration accumulation when there is a sustained large deviation, and further speed up the convergence process.

[0088] In the three control areas mentioned above, the proportional coefficient Integral coefficient and differential coefficients The value of generally increases from the fine-tuning region to the stable region and then to the high dynamic region, so that the control force of the controller adaptively increases as the intensity of the target's movement increases, and automatically weakens when the target tends to stabilize. Thus, under a single control framework, it takes into account both the control quality of fast response to large deviations and no overshoot for small deviations.

[0089] Based on the absolute value of the horizontal miss distance and the absolute value of the inter-frame change rate of the horizontal miss distance in the miss distance data set, the first target PID control parameter acting in the horizontal direction is determined by looking up the table in the two-dimensional parameter range table.

[0090] During processing, the horizontal miss distance in the miss distance dataset is first processed. Inter-frame variation rate of horizontal miss distance Take the absolute values ​​of each to obtain the absolute value of the horizontal miss distance. The absolute value of the inter-frame rate of change of horizontal miss distance Then, with As an index value for the dimension of the absolute value of the miss, As the index value of the absolute value of the rate of change dimension, a joint query is performed on the two-dimensional parameter level table. If... pixels and Pixels per second, then match the fine-tuning area parameter group; if pixels and pixels per second, then match the steady zone parameter group; if pixels and If the resolution is pixels per second, then the high dynamic range parameter group is matched. The preset PID control parameters corresponding to this region, determined by looking up the table, are the first target PID control parameters acting in the horizontal direction.

[0091] Based on the absolute value of the vertical miss distance and the absolute value of the inter-frame change rate of the vertical miss distance in the miss distance dataset, the second target PID control parameters acting in the vertical direction are determined by looking up the table in the two-dimensional parameter range table.

[0092] This step is independent of the horizontal table lookup process and is performed separately. Its input is the vertical miss distance from the miss distance dataset. Inter-frame variation rate of vertical miss distance During processing, first take the absolute value of both quantities to obtain the absolute value of the vertical miss distance. The absolute value of the inter-frame rate of change of the miss distance in the vertical direction Then, with As an index value for the dimension of the absolute value of the miss, Using the absolute value of the rate of change as an index value, a joint lookup is performed on the same two-dimensional parameter range table to match the corresponding control region. The preset PID control parameters corresponding to this region, determined by the table lookup, are the second target PID control parameters acting in the vertical direction. The target parameters for the horizontal and vertical directions are determined independently by looking up tables, enabling the control in both directions to adapt to different motion states and independently adjust the control intensity.

[0093] Specifically, determining a set of PID control parameters adapted to the current motion state based on the miss distance dataset further includes:

[0094] Obtain the first and second historical PID control parameters that were the final action of the previous control cycle.

[0095] Based on the first target PID control parameters and the first historical PID control parameters, an exponentially weighted moving average method is used for smoothing to obtain a set of PID control parameters that are finally applied to the horizontal direction control.

[0096] Based on the second target PID control parameters and the second historical PID control parameters, an exponentially weighted moving average method is used for smoothing to obtain a set of PID control parameters that ultimately act on the vertical direction control.

[0097] After determining the first target PID control parameters and the second target PID control parameters by looking up the two-dimensional parameter range table, the output is not directly used for the calculation of the gimbal rotation command. Instead, the following smoothing process is performed first.

[0098] First, obtain the historical parameters from the final action of the previous control cycle. The set of PID control parameters that ultimately acted on the horizontal direction control in the previous control cycle is recorded as the first set of historical PID control parameters; the set of PID control parameters that ultimately acted on the vertical direction control in the previous control cycle is recorded as the second set of historical PID control parameters. These two sets of historical parameters correspond to the smoothed proportional coefficients actually used for control law calculation in the horizontal and vertical directions in the previous control cycle, respectively. Integral coefficient and differential coefficients .

[0099] Then, the horizontal parameters are smoothed. The first target PID control parameter determined by the lookup table in the current control cycle and the first historical PID control parameter are weighted and fused using the exponentially weighted moving average method. The exponentially weighted moving average method is a time series smoothing method. Its basic operation logic is as follows: the target parameter value obtained from the lookup table in the current cycle is multiplied by a smoothing factor, the historical parameter value used in the previous cycle is multiplied by the complement of the smoothing factor, and the two are added together. The result is the set of PID control parameters that will ultimately act on the horizontal control in the current cycle. This smoothing factor determines the speed at which the final parameters converge to the target parameters: the larger the smoothing factor value, the faster the final parameters approach the target parameters, and the more agile the response; the smaller the smoothing factor value, the smoother the transition, but the slower the response. The specific value of the smoothing factor can be selected between 0 and 1, with a typical balanced value of 0.3.

[0100] Similarly, the parameters in the vertical direction are smoothed. The second target PID control parameters, determined by looking up a table in the current control cycle, and the second historical PID control parameters are weighted and fused using the same exponentially weighted moving average method described above, to obtain a set of PID control parameters that will ultimately affect the vertical control in the current cycle.

[0101] After the above smoothing process, the final output of the two sets of PID control parameters for horizontal and vertical control will serve as the direct input for calculating the gimbal rotation command in subsequent steps.

[0102] The technical contribution of this smoothing process lies in the fact that, during control region switching, parameters do not jump directly from one set of values ​​to another, but rather transition continuously and gradually. This eliminates abrupt changes in control values ​​caused by state region switching, effectively suppressing overshoot and mechanical oscillation in the gimbal servo system. Furthermore, the parameter smoothing processes in the horizontal and vertical directions are independent of each other, ensuring a smooth transition in control during their respective state changes without interference.

[0103] S6: Generate gimbal rotation command based on the miss distance dataset and the set of PID control parameters.

[0104] PID control algorithms are executed separately for the horizontal and vertical directions to calculate the corresponding gimbal rotation speed commands. The calculation logic of the PID control algorithm is to multiply the current deviation by a proportional coefficient. To obtain the proportional control component, multiply the cumulative value of the deviation over a past period by the integral coefficient. The integral control component is obtained, and the trend of the deviation is multiplied by the differential coefficient. The differential control components are obtained, and then the three components are added together to synthesize the total control output. For the horizontal direction, the horizontal miss distance is calculated. As the bias input, the inter-frame variation rate of the horizontal miss distance. As input for the trend of deviation change, in the horizontal direction , , As control parameters, the horizontal gimbal rotation speed command is calculated. For the vertical direction, the vertical miss distance is calculated. As the bias input, the inter-frame variation rate of the vertical miss distance. As input for the trend of deviation change, in the vertical direction , , As a control parameter, the vertical gimbal rotation speed command is calculated. The gimbal rotation speed command is a physical quantity that can directly affect the motor drive, such as voltage or pulse width modulation duty cycle.

[0105] The output of this step is a gimbal rotation speed command containing both horizontal and vertical rotation speed requirements. This command will serve as the direct basis for subsequent transmission to the gimbal drive module. The technical contribution of this step lies in the fact that, by integrating real-time dynamic miss distance data and adaptive PID control parameters, the generated rotation speed command can match the target's maneuvering state. It provides strong driving force for rapid correction during large deviations and reduces driving force to avoid overshoot during small deviations.

[0106] S7: Send the gimbal rotation command to the gimbal drive module to drive the gimbal to rotate so that the target returns to the center of the real-time image.

[0107] The communication connection is established between the processing unit that calculates the gimbal rotation commands and the gimbal drive module. The communication method uses a CAN bus with a baud rate of 1Mbps and a command transmission frequency of no less than 100Hz. The gimbal rotation speed commands received by the gimbal drive module include horizontal and vertical rotation speed commands, which are used to control the speed and direction of the pan-axis and tit-axis motors of the gimbal, respectively.

[0108] The servo controller inside the gimbal drive module converts received speed commands into current or voltage signals to drive the gimbal motor, causing it to rotate at a specified direction and speed. As the gimbal motor rotates, the camera fixed to the gimbal synchronously changes its shooting direction, thereby altering the target's imaging position in subsequent image frames. With the adjustment of the camera's pointing direction, the horizontal miss distance... The absolute value and vertical miss distance The absolute value of the target gradually decreases until the target approaches the center of the image. This process constitutes a complete visual feedback closed loop: the camera acquires images, the visual system detects and identifies the target and calculates the miss distance, the control law dynamically adjusts the PID parameters based on the miss distance and generates speed commands, the commands are sent to the gimbal drive module via the CAN bus, the gimbal rotates to change the camera's direction, which in turn affects the miss distance of the next frame image, and so on, until the miss distance approaches zero and the target stabilizes at the center of the real-time image.

[0109] Through high-frequency, low-latency real-time command issuance and a closed-loop tracking mechanism, the gimbal can follow high-speed maneuvering targets in real time and continuously lock the target near the center of the image, thereby achieving accurate and stable tracking of the target.

[0110] Specifically, a gimbal target tracking method based on miss distance also includes:

[0111] When the target detection model fails to output a bounding box of the same target identifier with a confidence level higher than the first confidence threshold in consecutive image frames with a first frame number threshold, it determines that the target is lost and enters the target loss processing mode.

[0112] In the target loss handling mode, the proportional coefficient in the current set of PID control parameters is reduced to a first preset percentage in the normal tracking mode, the derivative coefficient is reduced to a second preset percentage in the normal tracking mode, and the integral coefficient is set to zero.

[0113] In the target loss processing mode, the predicted position of the current frame is obtained based on the historical trajectory sequence before the target is lost. The virtual miss distance is calculated based on the predicted position of the current frame and the center coordinates of the real-time image. The virtual miss distance is used as the miss distance dataset to generate the gimbal rotation command.

[0114] The detection results output by the target detection model in each frame are evaluated. The evaluation logic is as follows: The detection results of multiple consecutive frames are examined to check if a bounding box exists in each frame that simultaneously satisfies the following two conditions: Condition 1, the target identifier carried by the bounding box is consistent with the identifier of the target being tracked; Condition 2, the confidence value of the bounding box is higher than a first confidence threshold. If no bounding box satisfying both conditions appears within a consecutive frame count threshold, the target is determined to be lost. The first frame count threshold specifies the maximum number of consecutive frames without valid detection required to determine loss. When the number of consecutive frames without valid detection reaches this threshold, the loss determination is valid. If a bounding box that simultaneously satisfies both identifier consistency and confidence value higher than the first confidence threshold is detected again in any frame during this period, the consecutive count is reset to zero, and the loss determination is invalid.

[0115] Upon determining that the target has been lost, the system immediately enters the target loss handling mode. This mode is a special operating state that differs from the normal tracking mode. In this mode, the system's control strategy, parameter configuration, and input data sources are all changed in order to attempt to maintain reasonable movement of the gimbal even when the target is not visible, and to create conditions for the target to reappear and be reacquired.

[0116] In the target loss handling mode, the proportional coefficient in the current set of PID control parameters is reduced to a first preset percentage in the normal tracking mode, the derivative coefficient is reduced to a second preset percentage in the normal tracking mode, and the integral coefficient is set to zero.

[0117] In normal tracking mode, the scaling factor Integral coefficient and differential coefficients The value is determined by looking up a two-dimensional parameter level table and smoothing it using an exponentially weighted moving average method. This value is used to generate rapid correction control force. After entering the target loss handling mode, the currently used proportional coefficient is adjusted. and differential coefficients Perform attenuation processing: adjust the scaling factor The value is reduced to fifty percent of the value taken in normal tracking mode; the differential coefficient is reduced. The value is reduced to fifty percent of the value taken in normal tracking mode. At the same time, the integral coefficient is... The value is set directly to zero.

[0118] proportionality coefficient The attenuation reduces the control loop's response strength to deviations, and the differential coefficients... The attenuation reduces the damping effect of the control loop, and the integral coefficient The zeroing of the parameters eliminates the continued impact of historical accumulated errors on the control output. Adjusting this parameter combination causes the gimbal to move in a smoother manner, following the predicted trajectory instead of actively trying to pull the target back to the center of the image, thus preventing overshoot or erroneous actions due to a lack of real visual feedback.

[0119] In the target loss processing mode, the predicted position of the current frame is obtained based on the historical trajectory sequence before the target is lost. The virtual miss distance is calculated based on the predicted position of the current frame and the center coordinates of the real-time image. The virtual miss distance is used as the miss distance dataset to generate the gimbal rotation command.

[0120] First, obtain the predicted position of the current frame. This predicted position is based on the historical trajectory sequence acquired before the target was lost, and is obtained through state estimation and extrapolation using a Kalman filter or other prediction algorithms. Its output is the predicted coordinates of the target in the horizontal direction. and vertical predicted coordinates This coordinate represents the location where the target is most likely to appear in the current frame.

[0121] Second, calculate the virtual miss distance. (This involves calculating the predicted horizontal coordinates.) Subtract the center x-coordinate of the real-time image This yields the virtual miss distance in the horizontal direction; the predicted coordinates in the vertical direction are then calculated. Subtract the center ordinate of the real-time image This yields the virtual miss distance in the vertical direction. This virtual miss distance is identical in form and meaning to the horizontal miss distance in the normal mode. and vertical miss distance They are consistent, both being pixel deviations of the target position relative to the image center.

[0122] Third, the virtual miss distances in the horizontal and vertical directions are used as the miss distance dataset for the current frame, replacing the miss distance dataset that should have been provided by real visual detection. This dataset is then input into the PID control algorithm to continue generating gimbal rotation commands.

[0123] Through this mechanism, when the target is obscured or disappears briefly, the system does not stop moving or search blindly. Instead, it performs evidence-based predictive tracking based on the target's historical movement patterns, keeping the gimbal pointing towards the area where the target may appear, thus creating favorable conditions for rapid re-acquisition when the target reappears.

[0124] Specifically, obtaining the current frame prediction position based on the historical trajectory sequence before the target was lost includes:

[0125] Obtain the historical target center point coordinate sequence before the target was lost, based on the last preset second frame number threshold.

[0126] The historical target center point coordinate sequence is input into a Kalman filter to obtain the state estimation vector output by the Kalman filter.

[0127] The predicted coordinates of the target in the horizontal direction and the predicted coordinates in the vertical direction are extracted from the first preset element position and the second preset element position of the state estimation vector, respectively, and used as the predicted position of the current frame.

[0128] First, the historical target center point coordinate sequence is obtained from the last set second frame threshold before the target was lost. This second frame threshold is ten frames, meaning the target center point coordinates are collected from each of the last ten images before the target was lost, forming a time series. This series records the target's positional trajectory just before its loss, serving as the data basis for subsequent state prediction.

[0129] Secondly, the historical target center point coordinate sequence is input into a Kalman filter. The Kalman filter is a well-known recursive estimation algorithm capable of estimating the internal state of a dynamic system based on a noisy observation sequence. In this technical solution, the Kalman filter uses a uniformly accelerated motion model to model the target motion, internally maintaining an eight-dimensional state vector. The state vector The elements included are, in order: the x-coordinate of the target center. Target center ordinate Velocity in the horizontal direction Velocity in the vertical direction acceleration in the horizontal direction Acceleration in the vertical direction Horizontal miss distance Vertical miss distance The Kalman filter uses a prediction-update cyclical iteration to adjust the state vector based on the input sequence of historical target center point coordinates. The estimation and correction processes are performed, and the final output is a posterior state estimation vector that integrates historical motion information and motion model constraints. .

[0130] Finally, from this state estimation vector Extract the predicted position from the state estimation vector. It is an eight-dimensional array, whose first preset element position corresponds to the first element of the vector, i.e., the x-coordinate of the target center. The estimated value; the position of the second preset element corresponds to the second element of the vector, i.e., the ordinate of the target center. The estimated values ​​are obtained. The values ​​of these two elements are directly extracted and used as the predicted coordinates of the target in the horizontal direction. and vertical predicted coordinates Together, these two elements constitute the predicted position for the current frame. This predicted position represents the image coordinates at which the target is most likely to appear in the current frame, and is used for subsequent calculations of the virtual miss distance.

[0131] In this way, even when the target is not visible, the system can generate an informed position prediction based on the target's historical movement patterns, providing key position information for predictive tracking and rapid re-acquisition.

[0132] Specifically, a gimbal target tracking method based on miss distance also includes:

[0133] In the target loss processing mode, the expanded search area is determined based on the predicted position of the current frame, and the size of the expanded search area is a first preset multiple of the target bounding box size of the last frame before the target was lost.

[0134] The expanded search area is used as the input area for the target detection model to perform target detection.

[0135] In the target loss handling mode, the target detection model originally performed target detection across the entire map. However, in order to recapture the target more quickly and reliably when it reappears, the detection range is reduced from the entire map to a local region centered on the predicted location, and the size of this region is enlarged to cover any possible errors between the predicted and actual locations.

[0136] First, the expanded search region is determined. This expanded search region is centered on the predicted position in the current frame, and its size is determined by multiplying the size of the target bounding box in the last frame before the target was lost by a first preset factor. The first preset factor is 1.8 times, meaning the width of the expanded search region is 1.8 times the width of the target bounding box in the last frame before the target was lost, and the height is 1.8 times the height of the target bounding box in the last frame before the target was lost. The purpose of this expansion factor is to provide a sufficiently large search range around the predicted position to accommodate any possible deviation between the Kalman filter prediction and the target's true position, while avoiding an excessively large search range that introduces too much irrelevant background interference or increases unnecessary computational overhead.

[0137] Subsequently, the expanded search area is set as the input region for the target detection model. In the next frame and subsequent image frames during the target loss handling mode, the target detection model performs target detection only within this expanded search area, rather than across the entire image. The confidence threshold of the target detection model is also adjusted accordingly, with a recapture confidence threshold of 0.8, higher than the 0.7 confidence threshold in normal tracking mode, to improve the reliability of recapture. If a bounding box with the same target identifier as the lost target and a confidence score higher than 0.8 is detected within the expanded search area, recapture is considered successful, the system exits the target loss handling mode, and returns to normal tracking mode.

[0138] By dynamically adjusting the position and size of the detection area, fast and reliable recapture is achieved when the target reappears, while avoiding the computational waste and false detection risks caused by blindly searching the entire image during the loss period.

[0139] Specifically, a gimbal target tracking method based on miss distance also includes:

[0140] When the target detection model re-detects the bounding box of the same target identifier with a confidence level higher than the second confidence threshold within the expanded search area under the target loss processing mode, it determines that the recapture is successful.

[0141] The attenuated proportional coefficient and differential coefficient are smoothly restored to their values ​​under the normal tracking mode, and the integral coefficient is restored to a non-zero value, thus exiting the target loss processing mode.

[0142] When the duration of the target loss processing mode exceeds a first duration threshold and recapture is still unsuccessful, the generation of the gimbal rotation command is stopped, and the gimbal is controlled to enter the scanning search state.

[0143] First, the criteria for successful recapture are clearly defined. In target loss handling mode, the target detection model only performs detection within the expanded search area. For each frame's detection results, it is determined whether a bounding box simultaneously satisfies the following two conditions: Condition 1, the target identifier carried by the bounding box is consistent with the target identifier of the lost target; Condition 2, the confidence value of the bounding box is higher than a second confidence threshold, which is 0.8. If a bounding box that simultaneously satisfies the above two conditions is detected in a frame, recapture is immediately determined to be successful. This second confidence threshold is higher than the confidence threshold of 0.7 used for detecting valid targets in normal tracking mode. Its technical significance lies in improving the reliability of recapture and avoiding false recapture caused by background interference or similar targets.

[0144] Second, the parameter restoration and mode exit operations after successful reacquisition are clarified. Upon successful reacquisition, the following processing is performed: the scaling factor currently in the attenuation state is adjusted... and differential coefficients The two coefficients are smoothly restored to their proper values ​​under normal tracking mode using an exponentially weighted moving average method, meaning they no longer remain at 50% decay but gradually recover to the target parameter values ​​under normal mode; the integral coefficient currently set to zero is... The parameters are restored to normal, non-zero values. Parameter restoration uses an exponentially weighted moving average method for a smooth transition, rather than instantaneous jumps, to avoid abrupt changes in gimbal speed commands caused by sudden parameter shifts, thus protecting the stability of the servo system. After parameter restoration is complete, the system exits the target loss handling mode, switches back to normal tracking mode, and regenerates gimbal rotation commands using the actual visual miss distance dataset.

[0145] Third, a timeout exit mechanism is clearly defined. If the duration of the target loss handling mode exceeds a first time threshold since its inception, and the conditions for successful re-acquisition are not met during this period, the target is determined to be permanently lost. The first time threshold is two seconds. In this case, the following actions are taken: the generation of gimbal rotation commands is stopped, and the gimbal motors are no longer controlled by the output of the tracking algorithm; the gimbal is controlled to enter a scanning search state, i.e., a periodic uniform speed scan of the entire image, in order to attempt to rediscover the target within a wider field of view.

[0146] It provides a complete decision-making logic for the system response after target loss: it achieves high success rate recapture by expanding the search area and using a high confidence threshold, ensures the smoothness of mode switching through smooth parameter recovery, and sets a clear timeout exit mechanism to prevent the system from performing invalid tracking indefinitely in the lost state. Thus, it achieves an optimal balance between recapture success rate after occlusion, system stability, and resource utilization efficiency.

[0147] Specifically, the different steps of the method are executed in parallel by multiple independent threads deployed on the same embedded platform, wherein:

[0148] The first thread performs the real-time image acquisition operation at a first frequency.

[0149] The second thread performs target detection and tracking operations and outputs the target bounding box.

[0150] The third thread executes the operations of generating the miss distance dataset, determining the set of PID control parameters, and generating the gimbal rotation command at a second frequency higher than the first frequency.

[0151] The second thread and the third thread interact with each other through a data buffer with a preset capacity. The third thread reads the latest target information from the data buffer through atomic operations. When the atomic operation fails to read the target information, the historical target information successfully read in the previous cycle is used to perform subsequent operations.

[0152] The entire approach is deployed on the same embedded platform, the NVIDIA Jetson Orin Nano, which integrates a graphics processor and a central processing unit. The different processing steps are distributed across multiple independent threads for parallel execution, forming an asynchronous pipeline to improve system throughput and real-time responsiveness.

[0153] The first thread is the image acquisition thread, which performs real-time image acquisition operations at a first frequency of thirty frames per second, matching the camera's frame rate. This thread acquires video streams from the camera mounted on the gimbal and outputs real-time image frames in YUYV format with a resolution of 1920 pixels by 1080 pixels. The scheduling policy for the first thread is configured as SCHED_OTHER, which is a conventional non-real-time scheduling policy.

[0154] The second thread is the object detection and tracking thread, which performs object detection and tracking operations. This thread receives real-time images acquired by the first thread, runs a YOLOv8n object detection model accelerated by TensorRT for inference, then associates inter-frame object identifiers with the ByteTrack tracker, and finally outputs object bounding boxes with object identifiers. The scheduling policy of the second thread is also configured as SCHED_OTHER.

[0155] The third thread is the control law calculation thread, which performs operations such as generating the miss distance dataset, determining a set of PID control parameters, and generating gimbal rotation commands at a second frequency. The second frequency is 100 Hz, higher than the first frequency of 30 frames per second. The technical significance of this frequency selection is that the control law calculation cycle is shorter than the image acquisition cycle, allowing for multiple updates of control commands between two image frames, ensuring the response speed and command continuity of the control loop. The scheduling strategy for the third thread is configured as SCHED_FIFO, with a priority of 99, which is a real-time scheduling strategy to ensure that the control law calculation task can promptly preempt processor resources.

[0156] Data interaction between the second and third threads is conducted through a data buffer. This data buffer is a lock-free circular buffer with a preset capacity of one, meaning it stores only the latest target information for one frame. The write and read synchronization mechanism employs atomic operations using mutex locks: the second thread, acting as the producer, attempts to acquire the mutex lock before writing data. If successful, it writes the latest target detection result to the buffer and releases the lock. The third thread, acting as the consumer, attempts to acquire the same mutex lock at a frequency of 100 Hz. If successful, it reads the data from the buffer and releases the lock; the read data is the latest target information. If the third thread fails to acquire the lock, it indicates that the second thread is not writing new data or is currently writing. The third thread does not block and wait but instead uses the historical target information successfully read in the previous control cycle to perform subsequent operations. This zero-order hold strategy ensures that the third thread can continuously output control commands at a frequency of 100 Hz, preventing interruption or jitter of control commands even with a low image detection frame rate or occasional delays.

[0157] This multi-threaded architecture assigns three types of operations with significantly different time consumption—image acquisition, visual reasoning, and control calculation—to independent threads of different priorities for parallel processing. It also achieves efficient and non-blocking data synchronization between threads through a lock-free buffer with a capacity of one, keeping the end-to-end processing latency within 120 milliseconds, thus meeting the real-time requirements of high-dynamic target tracking on embedded platforms.

[0158] Specifically, the third thread and the fourth thread used to send the gimbal rotation command are configured with a first scheduling strategy and a first priority.

[0159] The first thread and the second thread are configured with the second scheduling policy and the second priority.

[0160] The value of the first priority is higher than the value of the second priority.

[0161] The system has four independent threads. The fourth thread is the command sending thread, which sends the gimbal rotation commands generated by the third thread to the gimbal driver module via the CAN bus at a frequency of no less than 100 Hz. The third and fourth threads together constitute the real-time critical path for control command generation and issuance.

[0162] Threads 3 and 4 are configured with the first scheduling policy and the first priority. The first scheduling policy is SCHED_FIFO, a first-in-first-out real-time scheduling policy; the first priority value is 99. Threads 1 and 2 are configured with the second scheduling policy and the second priority. The second scheduling policy is SCHED_OTHER, a time-sharing non-real-time scheduling policy; the second priority value is below 99.

[0163] The higher priority value of the first priority means that the third and fourth threads, configured with the SCHED_FIFO scheduling strategy, have a higher preemption priority than the first and second threads when the operating system schedules tasks. When the third or fourth thread is ready, the operating system will prioritize its execution, while the first and second threads will only receive processor time when the third and fourth threads are not running. This priority configuration ensures that the control law calculation and instruction sending tasks, which have the highest real-time requirements, can obtain deterministic response times and stable execution cycles, unaffected by time-consuming tasks such as image acquisition and target detection, thus guaranteeing the real-time performance and determinism of the entire closed-loop control system.

[0164] In one specific embodiment, the method of the present invention is deployed on an NVIDIA Jetson Orin Nano embedded platform for real-time tracking of high-speed moving targets in intelligent security scenarios. The platform integrates a 1024-core Ampere architecture graphics processor and a 6-core ARM architecture central processing unit. The camera uses an IMX415 sensor, fixed on a two-axis gimbal, and outputs a real-time video stream in 1920×1080 pixel YUYV format.

[0165] When S1 is executed, the first thread acquires real-time image frames from the camera at a frequency of 30fps. The scheduling policy for this thread is SCHED_OTHER.

[0166] During execution of S2, the second thread receives real-time image frames and calls the YOLOv8n object detection model accelerated by TensorRT for inference. The model input size is adjusted to 640 pixels × 640 pixels, and the inference results are post-processed with non-maximum suppression, where the intersection-over-union threshold is set to 0.45 and the confidence threshold is set to 0.7. The post-processed results are sent to the ByteTrack tracker, whose trajectory buffer is set to 30 frames, and the output is a target bounding box with an object identifier, with coordinates at the top left corner. bottom right corner The second thread's scheduling strategy is also SCHED_OTHER.

[0167] When executing S3, calculate the coordinates of the target center. ,in , Image center coordinates ,540). Calculate the horizontal miss distance. and vertical miss distance and their inter-frame change rates and To facilitate consistent expression in subsequent steps, when the horizontal and vertical directions are not distinguished and only the general properties of the miss distance are discussed, the symbol will be used. Generally refers to the amount of missed target, expressed in symbols. Generally refers to the rate of change of the miss distance. This is used when determining control parameters for the horizontal and vertical directions separately. and These refer to the miss distance and its rate of change in the corresponding directions, i.e., for the horizontal direction... = , = For the vertical direction = , = .

[0168] When executing S4, the result obtained from S3 will be used. , , , The off-target data set is aggregated into the current frame.

[0169] When executing S5, a two-dimensional parameter level table is first constructed and stored in memory. This table uses the absolute value of the miss distance and the absolute value of the rate of change as a joint index, and contains three control areas: the fine-tuning area ( pixels and (pixels per second), corresponding , , Stable area ( pixels and (pixels per second), corresponding , , High dynamic zone ( pixels and (pixels per second), corresponding , , Then, respectively with and The first target PID control parameters are obtained by looking up the table. and The second target PID control parameters are obtained by looking up the table. Finally, the exponentially weighted moving average method is used, with a smoothing factor of 0.3, to weight and fuse the first and second target PID control parameters with the historical parameters of the previous cycle, respectively, to obtain the final coefficient values ​​used for control.

[0170] When executing S6, with and As input, the horizontal gimbal rotation speed command is calculated using the final PID control parameters in the horizontal direction; and As input, the vertical gimbal rotation speed command is calculated using the final PID control parameters in the vertical direction.

[0171] When executing S7, the speed command is sent to the gimbal drive module via the CAN bus at a baud rate of 1Mbps and a frequency of 100Hz, driving the Pan axis and Tilt axis motors to rotate, so that the target returns to the center of the image.

[0172] S3 to S6 are executed by the third thread at a frequency of 100Hz, while the transmission of S7 is executed by the fourth thread. Both the third and fourth threads use the SCHED_FIFO scheduling policy with a priority value of 99. Data is transferred between the second and third threads via a lock-free circular buffer of capacity one. The third thread attempts to read data atomically each time; if successful, it uses the latest target information; if it fails, it uses the data from the previous cycle, ensuring the continuity of control commands.

[0173] Based on a specific embodiment, this embodiment further describes the handling after the target is briefly lost. If the target is obscured by an obstacle during tracking, the following mechanism is executed.

[0174] The condition for determining target loss is: the target detection model does not output a bounding box with a confidence level higher than the first confidence threshold of 0.7 that matches the current target identifier for three consecutive frames. If a valid bounding box is detected again before the number of consecutive frames reaches three, the counter is reset to zero. After the condition is met, the target loss processing mode is entered.

[0175] After entering this mode, first set the scaling factor. and differential coefficients Attenuation to 50% of normal value, integral coefficient Set to zero. Next, obtain the historical target center point coordinate sequence of the last 10 frames before the target was lost, and input it into an eight-dimensional Kalman filter using a uniformly accelerated motion model. The process noise covariance matrix Q has a diagonal of [1,1,0.1,0.1,0.01,0.01,0.001,0.001], and the observation noise covariance matrix R has a diagonal of [5,5]. Extract the horizontal predicted coordinates from the first and second elements of the state estimation vector output by the filter. and vertical direction predicted coordinates .Will and As a virtual miss distance, it replaces the real miss distance in the S6 instruction generation. At the same time, with the predicted location as the center, the search area is expanded to 1.8 times the size of the bounding box before the loss, thus limiting the detection range of the object detection model.

[0176] In loss mode, if a bounding box with the same target identifier and a confidence level higher than the second confidence threshold of 0.8 is detected within the expanded region, recapture is considered successful. The decaying... , The system was smoothed back to normal values ​​using an exponentially weighted moving average method. Restore to a non-zero value and exit lost mode.

[0177] If the lost mode continues for more than 2 seconds without being recaptured, it is determined to be permanently lost, the generation of gimbal rotation commands is stopped, and the gimbal enters a full-map uniform speed scan search state.

[0178] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A gimbal target tracking method based on miss distance, characterized in that, Includes the following steps: Capture real-time images output by the camera mounted on the pan-tilt unit; Obtain the target bounding box output by the target detection model based on the real-time image; Based on the center coordinates of the target bounding box and the center coordinates of the real-time image, calculate the horizontal and vertical miss distances, and calculate the inter-frame variation rate of the horizontal and vertical miss distances; Based on the horizontal miss amount, the vertical miss amount, the inter-frame change rate of the horizontal miss amount, and the inter-frame change rate of the vertical miss amount, generate the miss amount dataset for the current frame. Based on the miss distance dataset, determine a set of PID control parameters that are adapted to the current motion state; Based on the miss distance dataset and the set of PID control parameters, a gimbal rotation command is generated; The gimbal rotation command is sent to the gimbal drive module to drive the gimbal to rotate so that the target returns to the center of the real-time image.

2. The gimbal target tracking method based on miss distance as described in claim 1, characterized in that, The step of determining a set of PID control parameters adapted to the current motion state based on the miss distance dataset specifically includes: A two-dimensional parameter level table is constructed by jointly indexing the absolute value of the miss distance and the absolute value of the rate of change. The two-dimensional parameter level table stores multiple control regions and a set of preset PID control parameters corresponding to each control region. Based on the absolute value of the horizontal miss amount and the absolute value of the inter-frame change rate of the horizontal miss amount in the miss amount data, the first target PID control parameter acting in the horizontal direction is determined by looking up the table in the two-dimensional parameter range table. Based on the absolute value of the vertical miss distance and the absolute value of the inter-frame change rate of the vertical miss distance in the miss distance dataset, the second target PID control parameters acting in the vertical direction are determined by looking up the table in the two-dimensional parameter range table.

3. The gimbal target tracking method based on miss distance according to claim 2, characterized in that, The step of determining a set of PID control parameters adapted to the current motion state based on the miss distance dataset further includes: Obtain the first and second historical PID control parameters that were the final action of the previous control cycle; Based on the first target PID control parameters and the first historical PID control parameters, an exponential weighted moving average method is used for smoothing to obtain a set of PID control parameters that are finally applied to the horizontal direction control. Based on the second target PID control parameters and the second historical PID control parameters, an exponentially weighted moving average method is used for smoothing to obtain a set of PID control parameters that ultimately act on the vertical direction control.

4. The gimbal target tracking method based on miss distance as described in claim 2, characterized in that, The multiple control areas in the two-dimensional parameter range table include a fine-tuning area, a stable area, and a high dynamic area; The preset PID control parameters corresponding to the fine-tuning zone satisfy the following: the proportional coefficient is less than the first proportional threshold, the integral coefficient is less than the first integral threshold, and the derivative coefficient is less than the first derivative threshold. The preset PID control parameters corresponding to the stable region satisfy the following: the proportional coefficient belongs to the second proportional range, the integral coefficient belongs to the second integral range, and the derivative coefficient belongs to the second derivative range. The preset PID control parameters corresponding to the high dynamic range satisfy the following: the proportional coefficient belongs to the third proportional range, the integral coefficient belongs to the third integral range, and the derivative coefficient belongs to the third derivative range. Wherein, the lower limit of the third proportional range is greater than the upper limit of the second proportional range, and the lower limit of the third differential range is greater than the upper limit of the second differential range.

5. The gimbal target tracking method based on miss distance according to claim 1, characterized in that, Also includes: When the target detection model does not output a bounding box of the same target identifier with a confidence level higher than the first confidence threshold in consecutive image frames of the first frame number threshold, it determines that the target is lost and enters the target loss processing mode. In the target loss handling mode, the proportional coefficient in the current set of PID control parameters is reduced to a first preset percentage in the normal tracking mode, the derivative coefficient is reduced to a second preset percentage in the normal tracking mode, and the integral coefficient is set to zero. In the target loss processing mode, the predicted position of the current frame is obtained based on the historical trajectory sequence before the target is lost. The virtual miss distance is calculated based on the predicted position of the current frame and the center coordinates of the real-time image. The virtual miss distance is used as the miss distance dataset to generate the gimbal rotation command.

6. A gimbal target tracking method based on miss distance as described in claim 5, characterized in that, The step of obtaining the current frame prediction position based on the historical trajectory sequence before the target was lost specifically includes: Obtain the historical target center point coordinate sequence before the target was lost, based on the last preset second frame number threshold. The historical target center point coordinate sequence is input into a Kalman filter to obtain the state estimation vector output by the Kalman filter. The predicted coordinates of the target in the horizontal direction and the predicted coordinates in the vertical direction are extracted from the first preset element position and the second preset element position of the state estimation vector, respectively, and used as the predicted position of the current frame.

7. A gimbal target tracking method based on miss distance as described in claim 5, characterized in that, Also includes: In the target loss processing mode, the expanded search area is determined based on the predicted position of the current frame, and the size of the expanded search area is a first preset multiple of the target bounding box size of the last frame before the target was lost; The expanded search area is used as the input area for the target detection model to perform target detection.

8. A gimbal target tracking method based on miss distance according to claim 7, characterized in that, Also includes: When the target detection model re-detects the bounding box of the same target identifier with a confidence level higher than the second confidence threshold within the expanded search area under the target loss processing mode, it determines that the recapture is successful. The attenuated proportional coefficient and differential coefficient are smoothly restored to their values ​​under the normal tracking mode, and the integral coefficient is restored to a non-zero value, thus exiting the target loss processing mode; When the duration of the target loss processing mode exceeds a first duration threshold and recapture is still unsuccessful, the generation of the gimbal rotation command is stopped, and the gimbal is controlled to enter the scanning search state.

9. A gimbal target tracking method based on miss distance as described in claim 1, characterized in that, The different steps of the method are executed in parallel by multiple independent threads deployed on the same embedded platform, wherein: The first thread performs the real-time image acquisition operation at a first frequency; The second thread performs target detection and tracking operations and outputs the target bounding box. The third thread executes the operations of generating the miss distance dataset, determining the set of PID control parameters, and generating the gimbal rotation command at a second frequency higher than the first frequency. The second thread and the third thread interact with each other through a data buffer with a preset capacity. The third thread reads the latest target information from the data buffer through atomic operations. When the atomic operation fails to read the target information, the historical target information successfully read in the previous cycle is used to perform subsequent operations.

10. A gimbal target tracking method based on miss distance as described in claim 9, characterized in that: The third thread and the fourth thread used to send the gimbal rotation command are configured with a first scheduling strategy and a first priority. The first thread and the second thread are configured with the second scheduling policy and the second priority; The value of the first priority is higher than the value of the second priority.

Images