Intelligent following shooting method based on optical flow method and target behavior prediction

Abstract

The present application relates to the pan-tilt control technical field, especially to a kind of intelligent follow-up shooting method based on optical flow method and target behavior prediction.The method includes the following steps: obtaining the motion target image sequence that pan-tilt camera continuously collects, the pixel displacement of adjacent frame image is calculated to form the optical flow field, wherein the optical flow field includes the optical flow vector of each pixel point in target area;Analysis of the motion disturbance structure characteristics of target area in optical flow field;According to motion disturbance structure characteristics, construct motion state vector, identify the emergency stop start state, direction change state and light mutation state of target through motion state vector;According to emergency stop start state, direction change state and light mutation state, generate pan-tilt control pre-adjustment instruction.The present application realizes the accurate identification of target comprehensive motion state through the analysis of divergence value, direction change characteristic value and brightness gradient inner product value, improves the reliability of follow-up shooting decision.

Description

Technical Field

[0001] This invention relates to the field of gimbal control technology, and in particular to an intelligent tracking method based on optical flow and target behavior prediction. Background Technology

[0002] Gimbal-mounted camera intelligent tracking technology is widely used in video surveillance, live sports broadcasts, security patrols, and drone aerial photography. Existing gimbal-mounted tracking systems primarily employ target detection and position feedback control methods. This involves detecting the target's positional deviation within the frame, calculating the gimbal compensation angle, and driving the gimbal to rotate and center the target within the frame. Typical tracking methods in existing technologies include predictive tracking based on Kalman filtering, probabilistic tracking based on particle filtering, and target detection and tracking based on deep learning. These methods mainly rely on the target's spatial position information for tracking decisions, employing a serial processing flow of "detection-calculation-execution."

[0003] However, in reality, the following problems often occur when using a gimbal for tracking shots: when the athlete stops suddenly, the optical flow suddenly drops to zero, and the gimbal will overshoot due to inertia; when the athlete is about to move from the sunlight area to the shadow area, the optical flow is normal at first, but it will often fail after 0.5 seconds due to the sudden change in lighting. Summary of the Invention Therefore, it is necessary for the present invention to provide an intelligent tracking method based on optical flow and target behavior prediction to solve at least one of the above-mentioned technical problems.

[0004] To achieve the above objectives, an intelligent tracking method based on optical flow and target behavior prediction includes the following steps: Step S1: Obtain the sequence of moving target images continuously captured by the gimbal camera, calculate the pixel displacement of adjacent frames to form an optical flow field, wherein the optical flow field contains the optical flow vector of each pixel in the target area; Step S2: Analyze the motion perturbation structure characteristics of the target region in the optical flow field; Step S3: Construct motion state vectors based on the structural characteristics of motion disturbances, and identify the target's sudden stop start state, change of direction state, and sudden change of illumination state through motion state vectors; Step S4: Generate PTZ control pre-adjustment commands based on emergency stop start status, direction change status, and sudden change in illumination status; Step S5: The gimbal controller executes the gimbal control pre-adjustment command to pre-adjust the gimbal motion parameters before the target motion state actually changes, so as to achieve synchronous tracking of gimbal motion and target motion.

[0005] This invention fuses three types of motion parameters—divergence, directional change characteristics, and the inner product of brightness gradients—into a motion state vector. Based on temporal changes and priority determination, it predicts sudden stops / starts, directional changes, and sudden changes in illumination of the target, thereby enabling pre-adjustment of gimbal control and significantly improving the tracking system's responsiveness to rapid movement and environmental changes. Specifically, on the one hand, the divergence measurement based on multi-point divergence sampling and weighted averaging accurately reflects the target's local convergence or divergence trends. Combined with the sign change and rate determination of the divergence sequence, it can identify the target's intention to move before it actually stops / starts and reduce tracking errors and jitter by adjusting the damping coefficient.

[0006] On the other hand, by using the difference in optical flow energy between the left and right sides of the region as a change-of-direction feature and examining the consistency of the sign and the rate of change over time, it is possible to distinguish between unilateral force application and abrupt changes of direction. This allows for the early generation of pre-turn and pre-rotation angle commands, reducing the gimbal compensation amplitude and latency, and maintaining stable image composition. Furthermore, by calculating the inner product of the brightness gradient using a fan-shaped prediction region starting from the centroid and combining it with distance-weighted summation, the boundary of sudden illumination changes can be located in advance, and the estimated arrival time can be estimated. This triggers exposure and feature switching strategies, avoiding tracking failures or feature loss due to sudden changes in illumination. Through the priority management of the predicted markers and the command encapsulation mechanism, the system can balance tracking stability and image quality in various sudden situations, enabling the gimbal to complete the coordinated adjustment of parameters such as damping, pre-rotation, and exposure before the target state changes. In summary, this method has significant advantages in improving tracking accuracy, reducing response latency, enhancing anti-interference capabilities, improving image quality, and enhancing system robustness and user experience. It is particularly suitable for real-time intelligent tracking applications in fast-moving targets and complex lighting scenarios. Attached Figure Description

[0007] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the steps of the intelligent tracking method based on optical flow and target behavior prediction of the present invention. Figure 2 This is a schematic diagram of divergence sampling and optical flow field according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the location of a sudden change in illumination boundary according to an embodiment of the present invention; Figure 4 This is a comparison chart of the predictive tracking effect of one embodiment of the present invention. Detailed Implementation

[0008] The technical method of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0009] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0010] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0011] To achieve the above objectives, please refer to Figures 1 to 4 This invention provides an intelligent tracking method based on optical flow and target behavior prediction, the method comprising the following steps: Step S1: Obtain the sequence of moving target images continuously captured by the gimbal camera, calculate the pixel displacement of adjacent frames to form an optical flow field, wherein the optical flow field contains the optical flow vector of each pixel in the target area; Step S2: Analyze the motion perturbation structure characteristics of the target region in the optical flow field; Step S3: Construct motion state vectors based on the structural characteristics of motion disturbances, and identify the target's sudden stop start state, change of direction state, and sudden change of illumination state through motion state vectors; Step S4: Generate PTZ control pre-adjustment commands based on emergency stop start status, direction change status, and sudden change in illumination status; Step S5: The gimbal controller executes the gimbal control pre-adjustment command to pre-adjust the gimbal motion parameters before the target motion state actually changes, so as to achieve synchronous tracking of gimbal motion and target motion.

[0012] Furthermore, the resolving characteristics of the optical flow field's motion perturbation structure include divergence values, directional eigenvalues, and the inner product of the brightness gradient. The calculation of the divergence values ​​includes: Multiple divergence sampling points are set within the target area to divide the target area into several sub-regions; In one embodiment, multiple divergence sampling points can be set within the target region according to a regular grid or an interest-based method. For example, the target region can be divided into grids with a fixed step size, with the center point of each grid serving as a divergence sampling point. To avoid overly sparse sampling that makes it difficult to extract local motion trends, the sampling point density can be adaptively adjusted according to the size of the target region; the larger the target region, the more sampling points are required.

[0013] For example, for a target area with a boundary of 80×80 pixels, a divergence sampling point can be set every 10 pixels to form a total of 64 sampling points in 8×8; at the same time, the target area is divided into 64 corresponding sub-regions, and each sub-region corresponds to a sampling point.

[0014] For each divergence sampling point, extract the set of optical flow vectors around that sampling point, and calculate the rate of change of the horizontal component of the optical flow vector along the horizontal direction and the rate of change of the vertical component along the vertical direction. In one embodiment, for each divergence sampling point, a set of optical flow vectors can be extracted from its surrounding preset neighborhood window (e.g., 3×3 or 5×5 pixels). Each optical flow vector contains a horizontal component (u) and a vertical component (v). The spatial rate of change of the horizontal component u with respect to the x-direction is then calculated. and the spatial variation rate of the vertical component v with respect to the y-direction .

[0015] For example, for optical flow data in a 3×3 neighborhood, the derivative can be approximated using the central difference method, for example... If the u-component of a pixel to the right of the sampling point is 2.5 and to the left it is 1.0, then the horizontal rate of change is approximately (2.5 − 1.0) / 2 = 0.75. The vertical rate of change can be calculated similarly.

[0016] Add the rate of change in the horizontal direction to the rate of change in the vertical direction to obtain the local divergence value of the divergence sampling point; In one embodiment, the local divergence values ​​can be obtained by direct addition, i.e. This value reflects whether the optical flow field around the sampling point is diverging or converging.

[0017] For example, if a certain sampling point is calculated to receive... =0.75, and If the local divergence is -0.20, then the local divergence value is 0.55, indicating a slight divergence trend in the region, meaning that the target's local motion is expanding outward. If the local divergence is negative, it indicates convergence, which may represent a sudden deceleration or near-stationary movement of the target.

[0018] The local divergence values ​​of all divergence sampling points are weighted and averaged to obtain the divergence value of the target region. Specifically, the weighted average is such that the sampling points located in the center of the target region have a higher weight than the sampling points located in the edge region.

[0019] In one embodiment, a weighting function can be set based on the distance between the sampling point and the center of the target region, for example, using a linear decay or Gaussian decay mode, to give a higher weight to the central region. The weighted average can be expressed as... .in For the first Local divergence values ​​at each sampling point For the corresponding weights.

[0020] For example, for an 8×8 sampling point array, the weight of the central 4×4 region can be set to 1.0, and the weights of the surrounding concentric regions can be set to 0.8, 0.6, 0.4, and 0.2 at the edges, respectively. If the divergence of the center point is generally 0.8 in a certain frame, while that of the edge points is 0.3, then the final weighted divergence is more inclined to reflect the movement trend of the central region.

[0021] See Figure 2 This demonstrates the distribution of divergence sampling points and the overall structure of the optical flow field within the target area. In the camera view, the target area is marked by a red dashed box, and this area is divided into a left sub-region and a right sub-region by a central dividing line. Multiple divergence sampling points (marked as P1 to P9 with orange dots) are set up within the target area. These sampling points are distributed in a regular grid to collect local optical flow information.

[0022] Optical flow vectors (indicated by green arrows) exist around each sampling point, reflecting the motion trend of pixels within the target region. The central sampling point P5 is located at the centroid of the target, and its optical flow vector (thick black arrow) represents the main direction of motion of the target. It can be observed in the figure that the optical flow vector distribution differs between the left sub-region (containing P1, P4, and P7) and the right sub-region (containing P3, P6, and P9). This asymmetry can be used to calculate the directional feature value.

[0023] The blue arrows around the camera's view indicate the background optical flow field, providing reference information for environmental motion. The legend labels the optical flow vector (green arrow), divergence sampling points (orange dots), target region (red dashed box), and center boundary line (orange dashed line) to facilitate understanding of each element. The coordinate system shows the X-axis as horizontal and the Y-axis as vertical.

[0024] Furthermore, the calculation of the directional eigenvalue includes: Divide the target area into a left sub-region and a right sub-region using the center line of the target area as the boundary; In one embodiment, the smallest bounding rectangle of the target region in the image can be determined first, and then its center position can be calculated based on the horizontal width of the rectangle. This center position serves as a vertical reference line to divide the target region into two equal-width sub-regions.

[0025] For example, when the detected target region is 80 pixels wide, the range of horizontal coordinates from 0 to 40 can be designated as the left region, and the range from 40 to 80 as the right region. This division allows the system to spatially distribute the target's overall motion information across the two regions, making subsequent motion asymmetry detection clearer.

[0026] For each pixel in the left and right sub-regions, calculate the magnitude of its optical flow vector and square it. Sum the squared optical flow magnitudes of all pixels on the left to obtain the optical flow energy values ​​on the left and right. In one embodiment, the system iterates through every pixel in the left region, reading the optical flow vector for each pixel. This optical flow vector consists of horizontal and vertical displacement components. To represent the motion intensity of the pixel, the horizontal and vertical components are multiplied by themselves, and then the two multiplication results are added together. This can be described as "adding the displacement contributions in two directions." The resulting value can be considered the motion energy of that pixel. The system then sequentially accumulates the motion energy of all pixels in the left region to obtain the left optical flow energy value. The same processing method is used for the right region to obtain the right optical flow energy value.

[0027] For example, if there are more pixels with obvious movement in the left region, the accumulated optical flow energy may reach 220, while the accumulated energy in the right region is only 140, indicating that the movement on the left is more intense.

[0028] The difference between the optical flow energy values ​​on the left and right sides is calculated and normalized, then used as the directional feature value of the target region.

[0029] In one embodiment, a difference can be obtained by subtracting the energy on the right side from the energy on the left side, which represents the difference in motion of the target in the left and right directions. A positive difference indicates that the motion on the left side is stronger, and a negative difference indicates that the motion on the right side is stronger. To avoid the difference being too large due to excessively high overall motion speed, which could affect the comparison between different frames, the system can further proportionally convert this difference to the sum of the left and right energies, so that the final direction change feature value is kept within a fixed range, thus ensuring a consistent judgment criterion.

[0030] For example, when the energy value on the left is 220 and the energy value on the right is 140, the difference is 80. After proportional conversion with the total energy of 360, a change-of-direction characteristic value of approximately 0.22 can be obtained. The positive sign indicates that the movement on the left is stronger, representing that the target may be turning to the right.

[0031] Of particular importance is that the calculation of the difference between the optical flow energy values ​​on the left and right sides, and the normalization process, is used as the directional feature value of the target region as follows: Calculate the difference between the optical flow energy values ​​on the left and right sides to obtain the original energy difference; The total energy value is obtained by summing the optical flow energy values ​​on the left and right sides. Divide the original energy difference by the total energy value to obtain the energy difference ratio. The total number of pixels in the target area is counted, and the number of pixels in the left sub-region and the right sub-region are counted separately. The ratio of the number of pixels in the left and right sub-regions is calculated. Divide the energy difference ratio by the ratio of the number of left and right pixels to eliminate the influence of the asymmetry of the left and right regions caused by the displacement of the target in the image, and obtain the normalized directional feature value. The physical meaning of normalization is to transform absolute energy difference into relative energy density difference, so that the directional characteristic value is not affected by the specific position and size of the target in the picture, but only reflects the asymmetry of the target's left and right movement.

[0032] Furthermore, the calculation of the inner product value of the brightness gradient includes: Extract the optical flow vector of the centroid within the target region, and use the direction of the optical flow vector as the direction of motion; In one embodiment, the outline of the target region is determined in the current image frame, and the position of the region's centroid is obtained through pixel-by-pixel statistical methods, for example, by using the average of the horizontal and vertical coordinates of all pixels as the centroid coordinates. Subsequently, the optical flow vector in the optical flow field is obtained at the centroid position, and the direction of this vector is used as the main motion direction of the target.

[0033] For example, when the optical flow vector at the centroid has a component of approximately 3 in the lateral direction and approximately 1 in the longitudinal direction, the direction of motion can be considered to be roughly pointing to the upper right. The purpose of this step is to provide a unified directional reference for subsequent sector region prediction and brightness gradient projection.

[0034] Starting from the target centroid, a fan-shaped prediction region is formed by extending along the direction of motion. The angle range of the fan-shaped prediction region covers the deflection range to the left and right of the direction of motion. The extension radius of the fan-shaped prediction region is determined by the product of the magnitude of the optical flow vector and the preset prediction time window. In one embodiment, a deflection angle range can be set for the left and right sides of the direction of motion, for example, each set to approximately 15 degrees, so that the overall fan-shaped angle range can cover the possible future positional deviations of the target. The radius of the fan-shaped region can be linearly calculated based on the magnitude of the optical flow vector at the centroid and the prediction time window. For example, if the magnitude of the optical flow vector is approximately 4 and the prediction time window is set to 0.2 seconds, the radius can be set to approximately 0.8 pixel distance units.

[0035] For example, in a certain detection step, if the target moves to the upper right, a fan-shaped region with an angle of about 30 degrees and a radius of about 0.8 is expanded to the upper right with the centroid as the center, which is used to predict the relevant region of the background brightness gradient change.

[0036] The brightness spatial gradient field of the background image within the fan-shaped prediction area is calculated to obtain the brightness gradient vector of each pixel. In one embodiment, a brightness difference operation can be performed on all pixels within the sector area. For example, the brightness difference between adjacent pixels can be used to form an approximate brightness change trend, thereby generating a horizontal brightness change amount and a vertical brightness change amount for each pixel. The brightness gradient vector can be obtained by combining these two directions.

[0037] For example, if the brightness of a pixel is slightly dimmer on its left side than on its right side, while the difference in brightness between its top and bottom sides is small, then the brightness gradient of that pixel is mainly oriented to the right. Through continuous calculation, the brightness spatial structure within the entire sector can be obtained for subsequent dot product analysis.

[0038] For each pixel within the fan-shaped prediction region, calculate the dot product of the motion direction vector and the brightness gradient vector of that pixel to obtain the local inner product value; In one embodiment, since the motion direction vector at the target centroid and the brightness gradient vector of each pixel have been obtained, the corresponding directional quantities of the two can be multiplied and added point by point to form a local value representing the magnitude of the brightness gradient component in the motion direction.

[0039] For example, when the brightness gradient direction of a pixel is almost in the same direction as the motion direction, its inner product value is high; while when the brightness gradient direction is roughly perpendicular to the motion direction, its inner product value is close to zero.

[0040] The local inner product values ​​of all pixels are summed by distance weighting, with the weights decreasing linearly as the distance from the pixel to the centroid of the target increases, to obtain the brightness gradient inner product value.

[0041] In one embodiment, the weight of each pixel within the sector region can be determined based on its distance from the centroid. For example, pixels closer to the centroid have a higher weight, while pixels farther away have a weight that decreases linearly with increasing distance. Subsequently, the local inner product value of each pixel is multiplied by its corresponding weight and accumulated to obtain the brightness gradient inner product value of the entire region.

[0042] For example, the pixel weight 0.1 units away from the centroid can be set to 1, and the pixel weight 0.8 units away from the centroid can be set to 0.2. The two contribute differently to the final result, making the texture information closer to the center of motion more important.

[0043] Furthermore, step S3 includes the following steps: Step S31: Construct a motion state vector containing divergence values, directional feature values, and the inner product of brightness gradients based on the motion perturbation structure characteristics; In one embodiment, the divergence value, the directional feature value, and the brightness gradient inner product value have been obtained from the aforementioned steps, and the three are combined into a structured vector in a preset order, for example, using the divergence value as the first element, the directional feature value as the second element, and the brightness gradient inner product value as the third element, so that the motion state vector can fully express the local motion contraction trend of the target area, the degree of energy imbalance in the left and right directions, and the consistency of the brightness structure in the motion direction.

[0044] For example, in a certain frame of an image, the divergence value is about 0.6, the directional feature value is about 0.3, and the brightness gradient inner product value is about 0.4. Then, the motion state vector constructed in this embodiment can represent the current state of the target with certain contraction characteristics, small directional deviation, and relatively stable brightness structure.

[0045] Step S32: Determine whether the absolute value of the divergence value exceeds the divergence threshold, whether the absolute value of the directional feature value exceeds the curl threshold, and whether the inner product value of the brightness gradient satisfies the illumination change condition to obtain the prediction flag; In one embodiment, corresponding detection thresholds are set for different motion features. For example, a threshold for sudden stop detection can be set for the divergence value. When the absolute value of the divergence is greater than this threshold, the target is considered to have a significant tendency to contract. A curl threshold can be set for the direction change feature value to determine whether there is a significant imbalance of energy on the left and right sides. When the absolute value of the direction change feature exceeds this threshold, the target may be attempting to drastically adjust its direction. The brightness gradient inner product value is used to determine whether a sudden change in illumination has occurred. For example, when the inner product value is less than a certain illumination judgment reference value, the correspondence between the background brightness distribution and the direction of motion is considered to be weakened, and there may be an illumination anomaly.

[0046] For example, in a certain step, the divergence threshold is set to 0.5 and the curl threshold is set to 0.25. When the divergence value is 0.7, the directional feature value is 0.1, and the brightness gradient inner product value is 0.15, only the divergence triggers the corresponding condition, and an emergency stop prediction flag can be generated.

[0047] Step S33: Establish the priority order of the prediction markers, and identify the target's emergency stop start state, change of direction state, and sudden change of illumination state by using the motion state vector according to the priority order.

[0048] In one embodiment, a clear priority order is set for the three prediction flags to ensure that the system can still output the corresponding status prompts according to stable logic when multiple features are triggered simultaneously. For example, the emergency stop flag can be set to the highest priority, the change of direction flag to the medium priority, and the sudden change in illumination flag to the lowest priority. This ensures that when the target shows signs of contraction, regardless of whether it is accompanied by a change in illumination, it is preferentially identified as an emergency stop-related state. Subsequently, the system can sequentially read the feature values ​​in the motion state vector and combine them with the results of each threshold judgment to determine the final state classification.

[0049] For example, if at a certain moment both the divergence value exceeds the threshold and the brightness gradient inner product value is abnormal, the system will ultimately identify the state at that moment as an emergency stop start because the emergency stop flag has a higher priority, and will not misjudge it as an abnormal illumination.

[0050] Furthermore, step S32 includes: When the absolute value of the divergence exceeds the divergence threshold, the emergency stop start prediction flag is activated. For example, the target region divergence value calculated for each frame can be compared with a pre-set divergence threshold in the real-time processing module; if the divergence value at a certain moment is determined to be higher than the threshold in an absolute sense, an initial trigger flag for emergency stop start is generated and written into the prediction queue for subsequent timing verification.

[0051] When the absolute value of the change-of-direction feature exceeds the change-of-direction feature threshold, it is determined that the target area has unilateral force characteristics, and the change-of-direction prediction flag is activated. For example, the directional feature value originates from the asymmetry of optical flow energy in the left and right sub-regions. When its absolute value reaches or exceeds the threshold, the system records that moment as a possible unilateral force event in the directional module and combines the event with the sign consistency check of consecutive frames to confirm whether a directional trend truly exists.

[0052] When the inner product of the brightness gradient satisfies the illumination change condition, the illumination change prediction flag is activated.

[0053] For example, the inner product of the brightness gradient reflects the projection of the background brightness structure in the direction of motion. If this value drops below a certain lighting judgment reference value, or shows a significant trend of change in a short period of time, it is judged as a possible lighting change and the corresponding flag is activated to trigger the pre-adjustment of the exposure or feature switching strategy.

[0054] Furthermore, the emergency stop start prediction specifically refers to: Continuously store the divergence values ​​at multiple time points to establish a divergence value sequence; For example, during real-time operation, the system saves the divergence values ​​of the most recent N frames in chronological order to a circular buffer. The typical value of N can be 10 or 20, depending on the frame rate and the required time resolution. The subsequent emergency stop and start determination module will use this sequence as input to perform sign change and rate analysis.

[0055] The trend of sign change in the divergence value sequence is judged. When the divergence value changes from positive to negative, it is identified as the optical flow field changing from divergence to convergence, and it is determined to be an emergency stop state; when the divergence value changes from negative to positive, it is identified as the optical flow field changing from convergence to divergence, and it is determined to be an acceleration state. For example, the system detects the point in time when the sign changes from positive to negative in the divergence sequence and records this change as a transition from divergence to convergence, thus considering it as an emergency stop candidate; conversely, if the sign changes from negative to positive, it is recorded as a transition from convergence to divergence and considered as a start-up acceleration candidate.

[0056] The statistical divergence value sign remains consistent for a certain duration. When the duration exceeds the preset minimum reliable time threshold, the identification of the emergency stop start state is confirmed to be valid. For example, a time threshold T can be set for the duration. If the divergence remains above T continuously within the same symbol interval, the symbol trend is considered reliable. For example, if the system frame rate is 30 frames per second, T can be set to 3 frames or 0.1 seconds to avoid misjudgment caused by short-term disturbances.

[0057] It should be noted that the setting of T is strongly related to the application scenario; for highly dynamic sports tracking scenarios, T can be set shorter to improve responsiveness; for scenarios with a lot of background disturbance, T should be set longer to improve robustness.

[0058] Calculate the rate of change of the divergence value sequence over time. When the rate of change exceeds the threshold of drastic change, it is determined to be a strong emergency stop or a strong start state, and is used as an emergency stop / start state.

[0059] For example, the system can calculate the time average of the difference in divergence values ​​between adjacent frames to approximate the rate of change, and compare this average rate of change with a drastic change threshold; when the average rate of change increases rapidly and exceeds the threshold, the system marks the event as a strong emergency stop or strong start, prompting the gimbal to adopt a more aggressive damping or steering adjustment strategy.

[0060] Furthermore, the aforementioned direction prediction specifically includes: The direction of unilateral force application is determined by the sign of the directional characteristic value. When the optical flow energy value on the left is greater than that on the right, it is determined that the force is applied from the left; when the optical flow energy value on the right is greater than that on the left, it is determined that the force is applied from the right. In one embodiment, the optical flow energy on the left and right sides of the target area is collected and compared. The optical flow energy is the sum of the squared optical flow magnitudes of all pixels in both sides, thus reflecting the actual imbalance of motion information on the left and right sides. When determining the direction, only the total energy on both sides needs to be compared to obtain the directional judgment of unilateral force application.

[0061] For example, in a certain frame of an image, if the cumulative optical flow energy in the left region is about 1200 and the cumulative value in the right region is about 800, the system can directly determine that the target has a tendency to exert force on the left, and the corresponding change-of-direction feature value is positive; if the two are opposite, the change-of-direction feature value is negative, corresponding to force exertion on the right.

[0062] Perform a sign consistency check on the direction change feature values ​​of multiple consecutive frames. When the sign of the direction change feature values ​​remains consistent and the absolute value continuously exceeds the direction change feature threshold, the direction change trend determination is confirmed to be valid. In one embodiment, to prevent interference from single-frame energy deviation caused by environmental noise or local occlusion, the system records a short-time sequence of directional change feature values ​​and performs a consistency check on the sign of the sequence. If the feature value maintains the same sign for several consecutive frames and its absolute value is consistently higher than a preset directional change feature threshold, it indicates that unilateral force exertion is not accidental but a continuous trend, and thus can be formally recognized as a valid directional change.

[0063] For example, assuming the system frame rate is 30 frames per second, it is required that the feature values ​​have the same sign for at least 3 consecutive frames and significantly exceed the threshold. For example, if the directional feature values ​​in the three consecutive frames are 0.32, 0.41, and 0.36 respectively, then the system can confirm the existence of a valid leftward directional trend.

[0064] Calculate the rate of change of the change of direction characteristic value over time; determine the change of direction intensity based on the rate of change of time; when the rate of change of time exceeds the preset rapid change of direction threshold, it is identified as an abrupt change of direction, otherwise it is identified as a normal change of direction. For example, if the change-of-direction characteristic value in a certain frame is 0.20, and then instantly rises to 0.55 in the next frame, the system will determine that its rate of change over time is high. If this rate of change exceeds the set rapid change-of-direction threshold, such as 0.25, the system will identify this event as a sharp change of direction, indicating that the target is rapidly adjusting its direction to one side. Conversely, if the change is relatively slow, such as increasing from 0.20 to 0.28, it will be determined as a normal change of direction.

[0065] The direction change state is determined based on the direction of force applied on one side and the intensity of the change.

[0066] In one embodiment, the change-of-direction state is further classified by combining the aforementioned direction information and intensity level, so that the gimbal control module can formulate corresponding pre-adjustment strategies. For example, if it is determined that the force is applied to the left side and the change-of-direction intensity is a sharp change of direction, the system can mark the state as a strong left-side change of direction, prompting the control module to generate a pre-turn command with a larger amplitude in advance; if the force is applied to the right side but the rate of change is low, the system can determine it as a normal right-side change of direction, and the corresponding pre-adjustment amplitude is relatively gentle.

[0067] For example, in a certain detection, the optical flow energy on the left is significantly higher than that on the right, and the direction change feature value is positive and rapidly increases in multiple consecutive frames. Therefore, it is finally determined that the device is in a strong leftward direction change state at that moment.

[0068] Furthermore, the prediction of sudden changes in illumination specifically includes: The direction of light change is determined by the dot product of the brightness gradient. When the dot product of the brightness gradient is negative, it is identified that the target is about to enter the shadow area; when the dot product of the brightness gradient is positive, it is identified that the target is about to enter the bright area. In one embodiment, the system calculates the luminance gradient vector of the region containing the target centroid in the current frame and the next frame, and performs an inner product operation between this luminance gradient vector and the optical flow gradient vector to obtain the luminance gradient inner product value. Here, the luminance gradient represents the direction of pixel brightness change, while the optical flow gradient represents the projection direction of the target's motion direction onto the image plane. The system determines the direction of illumination change based on the sign of the luminance gradient inner product value: when the inner product value is negative, it means the luminance gradient direction is opposite to the target's motion direction, the target is moving towards a region of decreasing brightness, and it is predicted that it will enter a shadow region; when the inner product value is positive, it means the luminance gradient direction is consistent with the motion direction, the target is moving towards a region of increasing brightness, and it is predicted that it will enter a bright region. For example, if a target moves from a bright outdoor area towards a shadowed pillar, its optical flow direction points towards the pillar's shadow area, and the brightness in this area decreases significantly. Therefore, the luminance gradient direction is opposite to the optical flow direction, and the luminance gradient inner product value is negative. In this case, the system determines that the target is about to enter a shadow region.

[0069] Locate the illumination abrupt change boundary within the fan-shaped prediction region; measure the distance from the target centroid to the illumination abrupt change boundary, divide this distance by the magnitude of the optical flow vector of the target centroid, and calculate the estimated arrival time of the target to the illumination abrupt change boundary; In one embodiment, a fan-shaped prediction region is constructed with the target's centroid as the vertex and according to the target's current optical flow direction. This fan-shaped prediction region has a certain opening angle and prediction radius, used to define the area where the target may move and to accurately locate the boundary of illumination abrupt changes. Within this fan-shaped region, the location of the abrupt change in brightness gradient magnitude of each pixel is calculated, and the spatial boundary between bright and dark areas is detected, thereby determining the location of the illumination abrupt change boundary.

[0070] For example, when the target moves to the right in the monitoring screen, a fan-shaped area centered on the centroid and pointing to the right is constructed, and points of abrupt changes in brightness gradient values ​​are scanned within this area; if a brightness abrupt change band that is roughly perpendicular to the direction of the target's movement is detected, it is identified as the boundary of the illumination abrupt change.

[0071] Determine the urgency of the sudden change in light intensity based on the estimated arrival time; In one embodiment, based on the illumination abrupt change boundary located in the previous step, the shortest distance from the centroid to the illumination abrupt change boundary is obtained using a geometric distance calculation method (e.g., Euclidean distance) with reference to the target centroid coordinates. Subsequently, this distance is divided by the magnitude of the target centroid optical flow vector (i.e., the magnitude of the target's velocity on the image plane) to obtain the estimated time for the target to reach the illumination abrupt change boundary.

[0072] For example, if the distance from the target's centroid to the boundary of the illumination change is measured to be 15 pixels, and the current optical flow vector magnitude is 5 pixels / frame, then the estimated arrival time is 3 frames, meaning the target will enter the illumination change region approximately 3 frames later.

[0073] The state of light change is determined based on the direction of light change and the urgency of the light change.

[0074] In one embodiment, the urgency of sudden changes in illumination is assessed by classifying them according to the expected arrival time and a preset time threshold range. For example, the urgency can be divided into three levels: "high," "medium," and "low." When the expected arrival time is less than a first time threshold (e.g., 2 frames), it is determined to be of high urgency; when it is between two thresholds, it is determined to be of medium urgency; and when it exceeds a second threshold (e.g., 5 frames), it is determined to be of low urgency.

[0075] For example, if a prediction shows that the target will enter the shadow area after 1.5 frames, the system will classify the mutation event as "high urgency" to prompt subsequent modules to execute compensation or robust tracking strategies in advance.

[0076] Of particular importance is that locating the boundary of illumination abrupt change within the fan-shaped prediction region specifically involves: Within the fan-shaped prediction area, starting from the target centroid, sampling is performed point by point along the direction of motion with a fixed step size, and the brightness gradient magnitude of the surrounding local area is extracted at each sampling point. Local averaging of the brightness gradient magnitude at each sampling point is performed, and the moving average is calculated using the gradient magnitudes of the three sampling points before and after that sampling point to eliminate single-point noise interference. Iterate through the moving average of all sampling points, find the sampling point corresponding to the global maximum value of the moving average, and this sampling point is the initial location point of the illumination change boundary; Using the initial positioning point as the center, a sub-pixel-level fine search is performed within a step size range before and after it. By performing quadratic curve fitting on the brightness gradient magnitude, the peak position of the fitted curve is obtained as the precise position of the illumination change boundary. Calculate the Euclidean distance between the target's centroid coordinates and the precise location of the illumination change boundary, and use this distance as the actual distance from the target to the illumination change boundary.

[0077] See Figure 3 As shown, this illustrates the process of locating the boundary of a sudden change in illumination as a target moves from a bright area to a shadow area. The left side of the image represents the bright area (high-brightness background), and the right side represents the shadow area (low-brightness background). The boundary between the two is marked by a vertical black dashed line.

[0078] The target's centroid (orange dot) is located in the bright area, and its direction of motion (thick red arrow) points towards the shadow area. A fan-shaped prediction region (marked by a blue dashed line, approximately 30° in angle) is constructed along the direction of motion, with the target's centroid as its vertex. This fan-shaped region covers the spatial range the target may reach in the future. Within the fan-shaped prediction region, a sequence of sampling points (blue dots) is set along the direction of motion to detect changes in the brightness gradient point by point.

[0079] The peak point of the brightness gradient (yellow dot) is marked at the boundary of the sudden illumination change. This point corresponds to the location of the maximum value of the brightness gradient, i.e., the location where the distinction between light and dark is most obvious. The distance d from the centroid to the boundary is marked by a black dashed line, with a measured value of 350 pixels. Based on the optical flow velocity v = 5 pixels / frame, the system calculates the estimated arrival time T = d / v = 350 / 5 = 70 frames (approximately 2.3 seconds). This time parameter is used to determine the urgency of the sudden illumination change, thereby triggering exposure adjustment and feature switching strategies in advance.

[0080] Furthermore, step S4 includes the following steps: Step S41: Determine the damping adjustment mode based on the emergency stop start state and generate a gimbal damping coefficient adjustment command; In one embodiment, the adjustment mode of the gimbal damping is determined based on the emergency stop / start state identified in the previous step. If the target is in an emergency stop state, the damping coefficient of the gimbal is reduced to make the gimbal response more sensitive to follow the target's sudden stop movement; if the target is in a start-up / acceleration state, the damping coefficient is appropriately increased to smooth the gimbal rotation and avoid overshoot or jitter.

[0081] For example, when the monitored target suddenly stops, the PTZ damping coefficient is adjusted from a medium value to a low value, enabling the PTZ to stop moving quickly without lag. Subsequently, the calculated damping coefficient value is encapsulated into an instruction format, forming a PTZ damping coefficient adjustment instruction, which is executed by the downstream control module.

[0082] Step S42: Infer the target turning direction based on the unilateral force direction in the turning state and set the gimbal pre-rotation direction. Calculate the gimbal pre-rotation angle based on the turning intensity and the absolute value of the turning characteristic value, and generate the gimbal pre-rotation command. In one embodiment, the future direction of the target's change of direction is determined based on the direction of force applied on one side during the change of direction. For example, if force is applied from the left, the target may shift to the left, and if force is applied from the right, it may shift to the right. Then, by combining the change of direction intensity and the absolute value of the change of direction characteristic value, the angle by which the gimbal needs to rotate in advance is calculated to ensure that the gimbal's view is aligned with the target's direction of movement when the target changes direction.

[0083] For example, when a target turns right rapidly and has a large change-of-direction characteristic value, the gimbal needs to deflect 10 degrees in advance to follow the target. Finally, this deflection angle and direction are encapsulated into a gimbal pre-steering command and sent to the gimbal control module.

[0084] It should be noted that the pre-turn angle depends not only on the magnitude of the change-of-direction characteristic value, but can also be dynamically corrected by combining the target's movement speed and historical turning patterns to reduce over-pre-turn or following delay.

[0085] Step S43: Determine the feature switching strategy and exposure adjustment strategy by the unilateral force direction in the state of sudden illumination change, determine the pre-adjustment time parameter based on the urgency of the sudden illumination change, and generate feature switching and exposure adjustment instructions; In one embodiment, the switching strategy of the feature extraction algorithm is determined based on the unilateral force direction during a sudden change in illumination. For example, it switches to a high-contrast feature extraction mode when the target enters a shadow area and a brightness compensation mode when it enters a bright area. Simultaneously, a pre-adjusted time parameter is set according to the urgency of the illumination change to perform exposure adjustments in advance, ensuring that image acquisition is optimized before the illumination change occurs.

[0086] For example, when a target rapidly enters the building's shadow area and the urgency level is high, the image feature algorithm is switched 0.5 seconds in advance, and the exposure gain is increased to maintain image sharpness. Subsequently, these strategies are used to generate feature switching and exposure adjustment instructions to control the real-time adjustments of the image acquisition module and the algorithm module.

[0087] Step S44: Encapsulate the gimbal damping coefficient adjustment command, gimbal pre-steering command, and feature switching and exposure adjustment command into a gimbal control pre-adjustment command according to priority.

[0088] In one embodiment, the damping coefficient adjustment command, pre-steering command, and feature switching and exposure adjustment command are uniformly encapsulated according to a preset priority order to form a complete gimbal control pre-adjustment command. The priority order can be set according to the urgency of different operations; for example, the emergency stop start state adjustment has the highest priority, followed by the change of direction and following, and the illumination adjustment has the lowest priority.

[0089] For example, when a target suddenly accelerates and simultaneously enters a region of changing illumination, the system will first perform damping adjustment, then gimbal pre-rotation, and finally exposure and feature switching. The encapsulated gimbal control pre-tuning commands are sent to the gimbal execution unit through a unified interface to achieve coordinated control of mechanical rotation and image acquisition.

[0090] See Figure 4 The diagram illustrates the detection and prediction process of a gimbal-based intelligent tracking system when a target deviates from the center of the frame. The diagram includes four time-series scenes, corresponding to different processing states at times T1 and T2. (See reference...) Figure 4 In the upper part, at time T1 (left image), the target (a simple human figure) is located slightly to the left of the center of the image, moving to the right (indicated by the blue arrow). At this time, the target has not yet deviated significantly. At time T2 (right image), the target continues to move to the right, causing it to deviate from the center of the image. The system detects the target deviation and issues a red warning sign "Target Deviation!", prompting the gimbal to adjust to keep the target centered.

[0091] See Figure 4 The lower half demonstrates the system's predictive tracking mechanism. At the improved time T1 (lower left figure), the system detects optical flow convergence features through optical flow field analysis, activates the system's predictive module, and displays "System Detection: Optical Flow Convergence Feature" in the upper left corner of the interface. Based on the detected motion disturbance features, the system constructs a fan-shaped prediction region (blue dashed fan-shaped area) in front of the target's centroid to predict the target's future trajectory and possible deviation direction.

[0092] Therefore, the embodiments should be considered as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalents of the application are intended to be included within the invention.

[0093] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. An intelligent tracking method based on optical flow and target behavior prediction, characterized in that, Includes the following steps: Step S1: Obtain the sequence of moving target images continuously captured by the gimbal camera, calculate the pixel displacement of adjacent frames to form an optical flow field, wherein the optical flow field contains the optical flow vector of each pixel in the target area; Step S2: Analyze the motion perturbation structure characteristics of the target region in the optical flow field, where the motion perturbation structure characteristics include divergence value, directional characteristic value, and brightness gradient inner product value; Step S3: Construct a motion state vector based on the structural characteristics of the motion disturbance, and identify the target's sudden stop / start state, change of direction state, and sudden change in illumination state through the motion state vector; including: Step S31: Construct a motion state vector containing divergence values, directional feature values, and the inner product of brightness gradients based on the motion perturbation structure characteristics; Step S32: Determine whether the absolute value of the divergence value exceeds the divergence threshold, whether the absolute value of the directional feature value exceeds the directional feature threshold, and whether the inner product value of the brightness gradient satisfies the illumination change condition to obtain the prediction flag; Step S33: Establish the priority order of the prediction markers, and identify the target's emergency stop start state, change of direction state, and sudden change of illumination state through the motion state vector according to the priority order; Step S4: Generate PTZ control pre-adjustment commands based on emergency stop start status, direction change status, and sudden change in illumination status; Step S5: The gimbal controller executes the gimbal control pre-adjustment command to pre-adjust the gimbal motion parameters before the target motion state actually changes, so as to achieve synchronous tracking of gimbal motion and target motion.

2. The intelligent tracking method based on optical flow and target behavior prediction according to claim 1, characterized in that, The calculation of the divergence value includes: Multiple divergence sampling points are set within the target area to divide the target area into several sub-regions; For each divergence sampling point, extract the set of optical flow vectors around that sampling point, and calculate the rate of change of the horizontal component of the optical flow vector along the horizontal direction and the rate of change of the vertical component along the vertical direction. Add the rate of change in the horizontal direction to the rate of change in the vertical direction to obtain the local divergence value of the divergence sampling point; The local divergence values ​​of all divergence sampling points are weighted and averaged to obtain the divergence value of the target region. Specifically, the weighted average is such that the sampling points located in the center of the target region have a higher weight than the sampling points located in the edge region.

3. The intelligent tracking method based on optical flow and target behavior prediction according to claim 2, characterized in that, The calculation of the directional eigenvalue includes: Divide the target area into a left sub-region and a right sub-region using the center line of the target area as the boundary; For each pixel in the left and right sub-regions, calculate the magnitude of its optical flow vector and square it. Sum the squared optical flow magnitudes of all pixels on the left to obtain the optical flow energy values ​​on the left and right. The difference between the optical flow energy values ​​on the left and right sides is calculated and normalized, then used as the directional characteristic value of the target region.

4. The intelligent tracking method based on optical flow and target behavior prediction according to claim 3, characterized in that, The calculation of the inner product value of the brightness gradient includes: Extract the optical flow vector of the centroid within the target region, and use the direction of the optical flow vector as the direction of motion; Starting from the target centroid, a fan-shaped prediction region is formed by extending along the direction of motion. The angle range of the fan-shaped prediction region covers a deflection range of 15° to the left and right of the direction of motion. The extension radius of the fan-shaped prediction region is determined by the product of the magnitude of the optical flow vector and the preset prediction time window. The brightness spatial gradient field of the background image within the fan-shaped prediction area is calculated to obtain the brightness gradient vector of each pixel. For each pixel within the fan-shaped prediction region, calculate the dot product of the motion direction vector and the brightness gradient vector of that pixel to obtain the local inner product value; The local inner product values ​​of all pixels are summed by distance weighting, with the weights decreasing linearly as the distance from the pixel to the centroid of the target increases, to obtain the brightness gradient inner product value.

5. The intelligent tracking method based on optical flow and target behavior prediction according to claim 4, characterized in that, Step S32 includes: When the absolute value of the divergence exceeds the divergence threshold, the emergency stop start prediction flag is activated. When the absolute value of the change-of-direction feature exceeds the change-of-direction feature threshold, it is determined that the target area has unilateral force characteristics, and the change-of-direction prediction flag is activated. When the inner product of the brightness gradient satisfies the illumination change condition, the illumination change prediction flag is activated.

6. The intelligent tracking method based on optical flow and target behavior prediction according to claim 5, characterized in that, The emergency stop start prediction specifically refers to: Continuously store the divergence values ​​at multiple time points to establish a divergence value sequence; The trend of sign change in the divergence value sequence is judged. When the divergence value changes from positive to negative, it is identified as the optical flow field changing from divergence to convergence, and it is determined to be an emergency stop state; when the divergence value changes from negative to positive, it is identified as the optical flow field changing from convergence to divergence, and it is determined to be an acceleration state. The statistical divergence value sign remains consistent for a certain duration. When the duration exceeds the preset minimum reliable time threshold, the identification of the emergency stop start state is confirmed to be valid. Calculate the rate of change of the divergence value sequence over time. When the rate of change exceeds the threshold of drastic change, it is determined to be a strong emergency stop or a strong start state, and is used as an emergency stop / start state.

7. The intelligent tracking method based on optical flow and target behavior prediction according to claim 6, characterized in that, The specific meaning of the direction change prediction is as follows: The direction of unilateral force application is determined by the sign of the directional characteristic value. When the optical flow energy value on the left is greater than that on the right, it is determined that the force is applied from the left; when the optical flow energy value on the right is greater than that on the left, it is determined that the force is applied from the right. Perform a sign consistency check on the direction change feature values ​​of multiple consecutive frames. When the sign of the direction change feature values ​​remains consistent and the absolute value continuously exceeds the direction change feature threshold, the direction change trend determination is confirmed to be valid. Calculate the rate of change of the change of direction characteristic value over time; determine the change of direction intensity based on the rate of change of time; when the rate of change of time exceeds the preset rapid change of direction threshold, it is identified as an abrupt change of direction, otherwise it is identified as a normal change of direction. The direction change state is determined based on the direction of force applied on one side and the intensity of the change.

8. The intelligent tracking method based on optical flow and target behavior prediction according to claim 7, characterized in that, The prediction of sudden changes in illumination is specifically as follows: The direction of light change is determined by the dot product of the brightness gradient. When the dot product of the brightness gradient is negative, it is identified that the target is about to enter the shadow area; when the dot product of the brightness gradient is positive, it is identified that the target is about to enter the bright area. Locate the illumination abrupt change boundary within the fan-shaped prediction region; measure the distance from the target centroid to the illumination abrupt change boundary, divide this distance by the magnitude of the optical flow vector of the target centroid, and calculate the estimated arrival time of the target to the illumination abrupt change boundary; Determine the urgency of the sudden change in light intensity based on the estimated arrival time; The state of light change is determined based on the direction of light change and the urgency of the light change.

9. The intelligent tracking method based on optical flow and target behavior prediction according to claim 8, characterized in that, Step S4 includes the following steps: Step S41: Determine the damping adjustment mode based on the emergency stop start state and generate a gimbal damping coefficient adjustment command; Step S42: Infer the target turning direction based on the unilateral force direction in the turning state and set the gimbal pre-rotation direction. Calculate the gimbal pre-rotation angle based on the turning intensity and the absolute value of the turning characteristic value, and generate the gimbal pre-rotation command. Step S43: Determine the feature switching strategy and exposure adjustment strategy based on the direction of illumination change, determine the pre-adjustment time parameter based on the urgency of illumination change, and generate feature switching and exposure adjustment instructions; Step S44: Encapsulate the gimbal damping coefficient adjustment command, gimbal pre-steering command, and feature switching and exposure adjustment command into a gimbal control pre-adjustment command according to priority.

Images