Pan-tilt video group behavior prediction stability augmentation method and system

Abstract

The embodiment of the invention discloses a pan-tilt video group behavior prediction stability augmentation method and system, and the method comprises the steps: obtaining a video stream shot by a pan-tilt in real time, and carrying out the short-time group behavior prediction and long-time group behavior prediction of the video stream; according to the long-time prediction result, determining the probability that the group keeps static within a first preset time length in the future; when the long-time prediction confidence coefficient is higher than a first threshold value and the probability that the group keeps static is higher than a second threshold value, directly judging that an energy recovery stability augmentation condition is met; otherwise, judging that the energy recovery stability augmentation condition is met only when the short-time prediction confidence coefficient is higher than a third threshold value; under the condition that the energy recovery stability augmentation condition is met, the pan-tilt motor is controlled to enter a regenerative braking state to recover electric energy; under the condition that the energy recovery stability augmentation condition is not met, the mechanical motion amplitude of the holder is limited, and the image cutting proportion of electronic stability augmentation is increased for compensation stability augmentation. The power consumption is remarkably reduced and the low-frequency stability augmentation capability is improved while the picture is ensured.

Description

Technical Field

[0001] This application relates to the fields of image processing and PTZ control technology, and in particular to a method and system for predicting and stabilizing group behavior in PTZ video. Background Technology

[0002] Current gimbal stabilization technologies are mainly divided into two types: mechanical stabilization and electronic stabilization. Mechanical stabilization has a fast response but high power consumption and limited travel; electronic stabilization consumes almost no power but requires sacrificing image size and has weak ability to suppress large low-frequency jitter. The two usually use fixed weights or simple switching, and cannot intelligently allocate weights according to the actual shooting scene.

[0003] Meanwhile, existing crowd behavior prediction technologies are mostly used for security early warning or crowd flow statistics. Almost no one has deeply coupled their prediction results with gimbal motor control, and there is no solution to use the future static trend of the crowd to drive the gimbal regenerative braking and recover electrical energy.

[0004] Therefore, existing technologies generally suffer from high power consumption and difficulty in achieving both stabilization and image loss in crowded scenarios. Summary of the Invention

[0005] This application provides a method and system for predicting and stabilizing group behavior in PTZ video, which performs precise energy recovery and graded stabilization of the PTZ, significantly reducing power consumption and improving low-frequency stabilization capabilities while ensuring image quality.

[0006] This application provides the following solution: According to a first aspect, a method for stabilizing and predicting group behavior in gimbal video is provided. The method includes: acquiring a video stream captured in real time by a gimbal; performing short-term and long-term group behavior prediction on the video stream to obtain a short-term prediction result, a short-term prediction confidence level, a long-term prediction result, and a long-term prediction confidence level; determining the probability that the group will remain stationary within a first preset time period based on the long-term prediction result; directly determining that the energy recovery stabilization condition is met when the long-term prediction confidence level is higher than a first threshold and the probability that the group will remain stationary is higher than a second threshold; otherwise, further combining the short-term prediction confidence level for a second judgment, determining that the energy recovery stabilization condition is met only when the short-term prediction confidence level is higher than a third threshold; when the energy recovery stabilization condition is met, controlling the gimbal motor to enter a regenerative braking state to recover electrical energy, and using the reverse torque generated by the regenerative braking to mechanically stabilize the gimbal; when the energy recovery stabilization condition is not met, limiting the mechanical movement amplitude of the gimbal and increasing the image cropping ratio of electronic stabilization for compensation stabilization.

[0007] According to one achievable method in the embodiments of this application, the prediction duration of the short-term group behavior prediction is 0.2 seconds to 1 second, and the short-term prediction result and short-term prediction confidence are obtained by fast trajectory extrapolation based on optical flow and pedestrian detection boxes.

[0008] According to one achievable method in the embodiments of this application, the prediction duration of the long-term group behavior prediction is 4 to 15 seconds, and a lightweight temporal model based on a multi-head attention mechanism is used to obtain the long-term prediction result and the long-term prediction confidence.

[0009] According to one achievable method in this application embodiment, determining the probability that the group will remain stationary within a first preset time period based on the long-term prediction result includes: converting the predicted trajectory of each individual in the long-term prediction result within the first preset time period into a corresponding set of predicted displacement vectors; clustering the set of predicted displacement vectors to obtain at least one motion consistency cluster; calculating the proportion of individuals in the largest motion consistency cluster among the total number of people and the average displacement within that cluster; and using the sum of the products of the proportion and a first weight, the reciprocal of the average displacement within the cluster and a second weight as the probability that the group will remain stationary.

[0010] According to one achievable method in this application embodiment, the mechanical stabilization of the gimbal using the reverse torque generated by regenerative braking includes: after the gimbal motor enters the regenerative braking state, the actual reverse torque value generated by the motor during the regenerative braking process is collected in real time, and the reverse torque value is directly superimposed on the torque output terminal of the original closed-loop attitude control loop of the gimbal as a feedforward compensation amount; at the same time, the residual low-frequency jitter output by the gimbal angular velocity sensor is monitored, and when the direction of the residual low-frequency jitter is detected to be opposite to the direction of the current reverse torque, the regenerative braking current is instantaneously increased to actively amplify the reverse torque until the residual low-frequency jitter is suppressed to within a preset range.

[0011] According to one achievable method in an embodiment of this application, the compensation stabilization by limiting the mechanical movement amplitude of the gimbal and increasing the image cropping ratio of electronic stabilization includes: mapping corresponding first conservative component and second conservative component according to the long-term prediction confidence and the short-term prediction confidence respectively, wherein the conservative component increases monotonically as the corresponding confidence decreases; selecting the larger value of the first conservative component and the second conservative component as the final conservative coefficient; multiplying the original target tracking angular velocity of the gimbal by the final conservative coefficient to obtain the maximum allowable angular velocity actually output to the motor, while dynamically adjusting the image cropping ratio of electronic stabilization to a preset multiple proportional to the final conservative coefficient.

[0012] According to one of the embodiments of this application, the method further includes: after determining that the energy recovery stabilization conditions are met and before the gimbal motor enters the regenerative braking state, performing a slight pre-braking of the current attitude of the gimbal for 0.1 seconds to 0.4 seconds, so that the attitude of the gimbal is slightly offset by a preset small angle in the predicted average motion direction of the group, and then formally entering the regenerative braking state and applying a reverse torque.

[0013] According to a second aspect, a PTZ video group behavior prediction and stabilization system is provided. The system includes: a group behavior prediction unit configured to acquire a video stream captured in real-time by a PTZ camera, perform short-term and long-term group behavior prediction on the video stream, and obtain a short-term prediction result, a short-term prediction confidence level, a long-term prediction result, and a long-term prediction confidence level; a stationary probability determination unit configured to determine, based on the long-term prediction result, the probability that the group will remain stationary within a first preset time period in the future; and an energy recovery stabilization determination unit configured to determine, when the long-term prediction confidence level is higher than a first threshold and the group remains stationary... When the probability is higher than the second threshold, the energy recovery stabilization condition is directly determined to be met; otherwise, a second judgment is made based on the short-term prediction confidence level, and the energy recovery stabilization condition is only determined to be met when the short-term prediction confidence level is higher than the third threshold. The stabilization execution unit is configured to control the gimbal motor to enter the regenerative braking state to recover electrical energy when the energy recovery stabilization condition is met, and to use the reverse torque generated by the regenerative braking to mechanically stabilize the gimbal; when the energy recovery stabilization condition is not met, the mechanical movement amplitude of the gimbal is limited and the image cropping ratio of the electronic stabilization is increased for compensation stabilization.

[0014] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0015] According to a fourth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects.

[0016] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application hierarchically assesses the short-term and long-term group behavior prediction results and their confidence levels in gimbal video. Only when the group is highly likely to remain stationary and the prediction is highly reliable is regenerative braking activated to recover electrical energy, cleverly utilizing the reverse torque generated by braking to achieve low-frequency mechanical stabilization. When the prediction confidence is insufficient or the group is about to move violently, a conservative strategy is immediately adopted, limiting the mechanical movement amplitude and compensating by increasing the electronic stabilization cropping ratio. This solution, for the first time, directly transforms semantic-level prediction of group behavior into energy recovery and hierarchical stabilization decisions at the gimbal physical layer. While ensuring stable and smooth video, it significantly reduces the average power consumption of the gimbal in stationary group scenarios, improves low-frequency jitter suppression, and achieves seamless complementary switching between mechanical and electronic stabilization, resulting in a comprehensive technical effect of longer battery life and superior image quality.

[0017] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a PTZ video group behavior prediction and stabilization method provided in an embodiment of this application; Figure 2 A structural block diagram of the PTZ video group behavior prediction and stabilization system provided in the embodiments of this application; Figure 3 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0021] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0024] Figure 1 The flowchart of the PTZ video group behavior prediction and stabilization method provided in the embodiments of this application is as follows: Figure 1 As shown, the method may include the following steps: Step 101: Obtain the video stream captured in real time by the gimbal, and perform short-term group behavior prediction and long-term group behavior prediction on the video stream to obtain the short-term prediction result, short-term prediction confidence, long-term prediction result and long-term prediction confidence.

[0025] Step 102: Determine the probability that the group will remain stationary within the first preset time period in the future based on the long-term prediction results.

[0026] Step 103: When the long-term prediction confidence is higher than the first threshold and the probability of the population remaining stationary is higher than the second threshold, the energy recovery stabilization condition is directly determined to be met; otherwise, a second judgment is made in combination with the short-term prediction confidence, and the energy recovery stabilization condition is only determined to be met when the short-term prediction confidence is higher than the third threshold.

[0027] Step 104: If the energy recovery and stabilization conditions are met, control the gimbal motor to enter the regenerative braking state to recover electrical energy, and use the reverse torque generated by the regenerative braking to mechanically stabilize the gimbal; if the energy recovery and stabilization conditions are not met, limit the mechanical movement amplitude of the gimbal and increase the image cropping ratio of the electronic stabilization for compensation and stabilization.

[0028] As can be seen from the above process, this application hierarchically judges the short-term and long-term group behavior prediction results and their confidence levels in the gimbal video. Only when the crowd is highly likely to remain stationary in the future and the prediction is highly reliable, does it activate regenerative braking to recover electrical energy and cleverly utilize the reverse torque generated by braking to achieve low-frequency mechanical stabilization. When the prediction confidence is insufficient or the crowd is about to move violently, it immediately switches to a conservative strategy, limiting the mechanical movement amplitude and correspondingly increasing the electronic stabilization cropping ratio for compensation. This solution is the first to directly transform the semantic-level prediction of group behavior into energy recovery and hierarchical stabilization decisions at the physical layer of the gimbal. While ensuring stable and smooth images, it can significantly reduce the average power consumption of the gimbal in stationary crowd scenarios, improve the low-frequency jitter suppression capability, and achieve seamless complementary switching between mechanical and electronic stabilization, resulting in a comprehensive technical effect of longer battery life and better image quality.

[0029] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments. It should be noted that the terms "first" and "second" involved in this disclosure do not have limitations in terms of size, order, or quantity, but are only used to distinguish them in name. For example, "first threshold" and "second threshold" are used to distinguish two thresholds.

[0030] First, the above step 101, namely "acquiring the video stream captured in real time by the gimbal, performing short-term group behavior prediction and long-term group behavior prediction on the video stream respectively, and obtaining short-term prediction results, short-term prediction confidence, long-term prediction results and long-term prediction confidence", will be described in detail with reference to the embodiments.

[0031] When the gimbal is in operation, it continuously captures the surrounding scene using its onboard camera, creating a real-time video stream. This video stream is typically a high-definition image sequence of 30 to 120 frames per second, containing rich information about group motion. This invention first obtains this raw video stream directly from the gimbal's image signal processor or video encoding module to ensure that subsequent predictions are based on the latest and most complete image data, without loss of detail due to compression or latency.

[0032] For the same video stream, this invention simultaneously initiates predictions at two different time scales. Short-term group behavior prediction focuses on the group movement trend within the next 0.2 to 1 second. This prediction process typically employs a computationally low-cost optical flow method combined with a fast trajectory extrapolation method using pedestrian detection boxes, which can be completed within tens of milliseconds. The short-term prediction results include the predicted position and velocity of each person in the next few frames. Simultaneously, the model outputs a short-term prediction confidence score to characterize the impact of factors such as current illumination, occlusion, and density on the reliability of the short-term prediction. The higher the confidence score, the clearer and more reliable the short-term group movement trend.

[0033] Specifically, in each frame or every few frames, a lightweight pedestrian detection network (such as MobileNet-SSD, YOLOv5n, YOLOv8n, or even the lighter PP-PicoDet) is used to obtain the bounding boxes and corresponding detection confidence scores for all pedestrians in the current frame. The detection confidence scores in this step are directly inherited into the subsequent short-term prediction confidence scores. If a pedestrian's detection score is already low (occluded, at a distance, or turned to the side), the subsequent prediction confidence scores will also decrease accordingly.

[0034] After obtaining the pedestrian detection bounding box, this invention does not directly use a complex deep learning trajectory prediction model. Instead, it employs classic but highly efficient dense or sparse optical flow methods to estimate motion within the bounding box or key points. For example, a dense optical flow field is calculated for the entire image or the region of the detection bounding box, and then the median or weighted average of the optical flow vectors within each bounding box is taken as the instantaneous motion vector of the pedestrian. Alternatively, Shi-Tomasi corner points are extracted within the detection bounding box, or MediaPipe / AlphaPose is used to extract key points of the human body, and then these feature points are tracked to obtain a more accurate motion vector.

[0035] After obtaining the current speed of each person, the system adopts a simple but robust extrapolation strategy: if the speed change in the last 3 frames is less than the threshold, the constant speed model is used to extrapolate the trajectory for the next 0.2 to 1 seconds; if the speed fluctuation is large, the system switches to a first-order Kalman filter and uses the positions of the previous few frames as observations to predict the future position.

[0036] Simultaneously, this invention performs long-term group behavior prediction on the same video stream, with the prediction duration typically set between 4 and 15 seconds. This prediction requires capturing the interactions between individuals within the crowd, social forces, and overall movement intentions; therefore, a lightweight temporal model based on a multi-head attention mechanism is employed. The long-term prediction also outputs a complete trajectory sequence for each individual over a longer future period, along with a long-term prediction confidence score. This confidence score reflects the model's grasp of the distant future and is typically influenced by factors such as crowd density, behavioral complexity, and ambient lighting.

[0037] The core of the model is a multi-head attention layer, typically consisting of 4 to 8 attention heads. Each head independently learns different attention patterns: some heads focus on the repulsion and attraction relationships between nearby individuals, some focus on the consistency of group flow at a distance, and others capture sudden acceleration or turning intentions. Through multi-head parallel computation, the model can effectively avoid the gradient vanishing problem common in traditional recurrent neural networks within a prediction window of up to 15 seconds, while also modeling long-term dependencies in complex crowd scenarios more accurately. The attention weights naturally form an explicit representation of "who influences whom" during the computation process, making it more flexible and robust than traditional social force models.

[0038] To ensure deployment at the edge, this invention deeply simplifies the standard Transformer structure: the number of encoder layers is controlled to 2 to 4, with each layer having a hidden dimension of only 128 to 256; linear self-attention or Performers approximation is used to replace the standard quadratic complexity attention, making the computational cost increase linearly with the sequence length rather than quadratically; simultaneously, knowledge distillation is employed, using a teacher model with a larger number of parameters trained in the cloud, and then compressing the knowledge into the student model. The final model size is typically no more than 5MB, and the inference speed can reach over 120 frames per second on Qualcomm Snapdragon 8 series processors, fully meeting the real-time control requirements of gimbals.

[0039] The model ultimately outputs the complete trajectory sequence of each pedestrian for the next 4 to 15 seconds frame-by-frame through the decoder, which is the long-term prediction result. Simultaneously, a lightweight confidence head is appended after the final attention layer output. This head consists of two fully connected network layers, taking into account the statistical characteristics of the attention weight distribution, trajectory variance, and accumulated historical prediction errors, and outputting a scalar between 0 and 1 as the long-term prediction confidence. When crowd movement is highly consistent, historical trajectory linearity is high, and attention weights are concentrated, the confidence level approaches 1; when occlusion, sudden changes in lighting, or drastic changes in behavior patterns occur, the confidence level drops significantly, thus providing a reliable risk assessment basis for subsequent energy recovery decisions.

[0040] The following describes in detail step 102, namely "determining the probability that the group will remain stationary within a first preset time period in the future, based on the long-term prediction results", with reference to the embodiments.

[0041] The long-term prediction results contain the complete trajectory sequence of each individual within a first preset time period (typically 5 to 12 seconds). This invention first simplifies each trajectory into a single predicted displacement vector, that is, calculating the total displacement vector (including displacement magnitude and direction) of that individual from the current moment to the end of the first preset time period. In this way, all individuals are transformed into a set of displacement vectors. This transformation significantly reduces computational complexity while preserving the core information of each individual's future movement trend, facilitating subsequent cluster analysis.

[0042] As an implementable method, the present invention determines the probability that the group will remain stationary within a first preset time period based on the long-term prediction results by: converting the predicted trajectory of each individual in the long-term prediction results within the first preset time period into a corresponding set of predicted displacement vectors; clustering the set of predicted displacement vectors to obtain at least one motion consistency cluster; calculating the proportion of individuals in the largest motion consistency cluster among the total number of people and the average displacement within that cluster; and using the sum of the products of the proportion and a first weight, the reciprocal of the average displacement within the cluster and a second weight as the probability that the group will remain stationary.

[0043] Specifically, each trajectory is first simplified: the total displacement vector of the individual from the current moment to the end of the first preset time period is calculated, which is the difference between the starting position and the ending position, resulting in a two-dimensional or three-dimensional displacement vector containing the magnitude and direction of the displacement. In this way, all individuals are uniformly converted into a set of predicted displacement vectors.

[0044] Next, clustering is performed on the entire set of predicted displacement vectors. Common algorithms include density-based DBSCAN or improved K-means variants. The goal of clustering is to automatically group individuals with highly similar displacement vectors in magnitude and direction, thus forming at least one cluster of consistent motion. Typically, a near-stationary cluster with minimal displacement and several moving clusters with larger displacements in different directions will naturally emerge. This clustering process automatically ignores isolated noise points, ensuring that even localized agitation, scattering, or turning within the crowd does not affect the identification of the mainstream trend.

[0045] After clustering, the system identifies the largest cluster of individuals exhibiting consistent movement, known as the maximal cluster. This cluster typically represents the most prevalent movement pattern within the population. Two key indicators are then calculated: first, the proportion of individuals within the maximal cluster to the total number of individuals detected; a higher proportion indicates a more representative overall trend of stillness or consistent movement. Second, the average magnitude of all displacement vectors within the cluster, i.e., the average displacement within the cluster; a smaller value indicates a lower future movement amplitude and a greater likelihood of stillness in the main population. These two indicators together provide a quantitative description of the population's tendency towards stillness.

[0046] The probability of the group remaining stationary is calculated precisely as follows: multiply the proportion of individuals in the largest cluster by a first weight, and add the result of multiplying the inverse of the average displacement within the cluster by a second weight. The sum of these two weights is the final probability. The first and second weights can be preset in the actual system or adaptively adjusted according to the scenario. Typically, the first weight is set to 0.6 to 0.7 to emphasize the importance of subject consistency; the second weight is correspondingly set to 0.3 to 0.4 to ensure that displacement amplitude information is reasonably reflected. This weighted summation method stabilizes the probability value in the range of 0 to 1. When the proportion of individuals in the largest cluster is close to 100% and the average displacement within the cluster is close to 0, the probability is close to or equal to 1.

[0047] The following describes in detail step 103, namely, "when the long-term prediction confidence is higher than the first threshold and the probability of the population remaining stationary is higher than the second threshold, it is directly determined that the energy recovery stabilization condition is met; otherwise, a second judgment is made in conjunction with the short-term prediction confidence, and the energy recovery stabilization condition is only determined to be met when the short-term prediction confidence is higher than the third threshold."

[0048] First, the system checks whether the long-term prediction confidence level is higher than the first threshold, and whether the probability of the group remaining still is higher than the second threshold. The first threshold is typically set between 0.80 and 0.90, and the second threshold is set between 0.85 and 0.95. Only when both conditions are met simultaneously is the energy recovery stabilization condition directly determined to be satisfied. The core of this design lies in trusting the semantic-level understanding capability of the long-term prediction: when the long-term model is extremely confident in its grasp of the next 4 to 15 seconds and clearly judges that the main body of the crowd will soon be highly still, the system can boldly allow the gimbal motor to enter regenerative braking, using reverse torque to achieve near-zero power consumption low-frequency stabilization. This direct triggering path has the highest activation rate in typical scenarios such as the end of a concert or a stop during a parade, allowing the energy recovery efficiency to reach its peak.

[0049] If either the long-term prediction confidence level or the probability of stillness fails to meet the threshold, the system enters a secondary judgment, examining only whether the short-term prediction confidence level is higher than a third threshold. The third threshold is typically set between 0.88 and 0.95, higher than the first threshold, to reflect a more stringent conservative approach. Only when the short-term prediction confidence level is extremely high is the energy recovery stabilization condition ultimately deemed met. This fallback mechanism leverages the extremely high certainty of short-term predictions within 0.2 to 1 second. Even if long-term predictions lack confidence due to occlusion, sudden changes in illumination, or complex behavior, the system can still confirm in the short term that the population is "almost still at least now and for the next second," thus safely initiating regenerative braking.

[0050] In this invention, long-term predictions must simultaneously satisfy both the confidence level and the static probability to pass directly, reflecting the system's caution regarding the distant future. Short-term predictions, on the other hand, serve as a last line of defense, relying solely on a high confidence level for approval, as their prediction window is extremely short and the consequences of errors are controllable.

[0051] The following describes in detail step 104 above, namely, "when the energy recovery and stabilization conditions are met, the gimbal motor is controlled to enter the regenerative braking state to recover electrical energy, and the reverse torque generated by the regenerative braking is used to mechanically stabilize the gimbal; when the energy recovery and stabilization conditions are not met, the mechanical movement amplitude of the gimbal is limited and the image cropping ratio of the electronic stabilization is increased for compensation stabilization."

[0052] This application directly maps the semantic results of group behavior prediction to the physical control level of the gimbal motor: in high-confidence static scenarios, it actively pursues energy recovery and ultra-strong low-frequency stabilization; in scenarios with uncertain predictions, it immediately degenerates into a conservative strategy to ensure absolute image safety. This dual-mode switching mechanism achieves the best balance between power consumption, image quality, and low-frequency suppression capability, completely solving the three dilemmas of existing gimbals in crowd scenarios: "either high power consumption, large image loss, or severe low-frequency shaking."

[0053] Once the system determines that the energy recovery and stabilization conditions are met, the gimbal motor is immediately controlled to enter regenerative braking mode. In this state, the motor no longer actively outputs driving torque, but instead switches to generator mode, converting the kinetic energy generated by hand tremors or inertia of the gimbal into electrical energy, which is then fed back to the battery in real time or used for lens heating and defogging.

[0054] As an feasible approach, using the reverse torque generated by regenerative braking to mechanically stabilize the gimbal includes: after the gimbal motor enters the regenerative braking state, the actual reverse torque value generated by the motor during the regenerative braking process is collected in real time, and the reverse torque value is directly superimposed on the torque output terminal of the original closed-loop attitude control loop of the gimbal as a feedforward compensation; at the same time, the residual low-frequency jitter output by the gimbal angular velocity sensor is monitored, and when the direction of the residual low-frequency jitter is detected to be opposite to the direction of the current reverse torque, the regenerative braking current is instantaneously increased to actively amplify the reverse torque until the residual low-frequency jitter is suppressed to within a preset range.

[0055] Specifically, after the gimbal motor enters regenerative braking mode, the system immediately acquires the actual reverse torque value generated by the motor in real time through a high-precision current sensor or back EMF sampling circuit in the motor driver. This value is not a theoretical calculation value, but a real physical torque, usually read directly in Newton-meters. Subsequently, this invention uses this actual reverse torque value as a feedforward compensation amount, directly superimposed on the torque output terminal of the original closed-loop attitude control loop of the gimbal. This feedforward operation bypasses the integral delay of traditional PID control, enabling the reverse torque to participate in attitude correction instantly within milliseconds, significantly improving the system's response speed to low-frequency jitter, especially in the 0.1 Hz to 1 Hz hand tremor frequency band, almost achieving theoretical instantaneous suppression.

[0056] Meanwhile, the system continuously monitors the residual low-frequency jitter signal output by the gimbal's angular velocity sensor. Once the direction of the residual jitter is detected to be opposite to the current reverse torque direction, indicating that the existing braking torque is insufficient to completely counteract the disturbance, the system immediately sends a command to the motor driver to instantaneously increase the regenerative braking current. This operation proportionally amplifies the motor's reverse torque, typically increasing the torque intensity by 50% to 200% within 100 milliseconds. The system continuously monitors the residual jitter amplitude until it is suppressed to within a preset range (e.g., ±0.1 degrees per second) before gradually restoring the normal braking current. This closed-loop regulation transforms the reverse torque from passively generated to actively controllable, completely breaking the limitation of traditional regenerative braking being solely used for energy recovery.

[0057] When the system determines that the energy recovery stabilization conditions are not met, it will never risk activating regenerative braking. Instead, it immediately limits the mechanical movement range of the gimbal motors, typically multiplying the original target tracking angular velocity by a conservative coefficient less than 1, compressing the mechanical movement range to 40% to 70% of normal. Simultaneously, the system correspondingly increases the image cropping ratio of electronic stabilization, raising the original safe cropping margin from only 10% to 15% to 25% to 45%, using a larger cropping window to absorb residual jitter. This compensation strategy ensures that even under adverse conditions such as low prediction confidence and impending violent movement of the crowd, the image remains smooth and stable, avoiding sudden changes or loss of control caused by accidental triggering of regenerative braking.

[0058] As an implementable method, the compensation stabilization by limiting the mechanical movement range of the gimbal and increasing the image cropping ratio of electronic stabilization includes: mapping corresponding first and second conservative components based on the long-term prediction confidence and the short-term prediction confidence, respectively, wherein the conservative components monotonically increase as the corresponding confidence decreases; selecting the larger value of the first and second conservative components as the final conservative coefficient; multiplying the original target tracking angular velocity of the gimbal by the final conservative coefficient to obtain the maximum allowable angular velocity actually output to the motor, while dynamically adjusting the image cropping ratio of electronic stabilization to a preset multiple proportional to the final conservative coefficient.

[0059] Specifically, when the system determines that the energy recovery stabilization conditions are not met, it first performs independent mapping based on the current long-term and short-term prediction confidence levels to obtain the corresponding first and second conservative components. This mapping relationship is designed as a monotonically increasing function, meaning that the lower the confidence level, the larger the corresponding conservative component. A linear or piecewise linear mapping is typically used; for example, the conservative component is 0 when the confidence level is 1, and 0.6 when the confidence level is 0.5. This independent mapping method ensures that both the long-term uncertainty of the long-term prediction and the short-term uncertainty of the short-term prediction can independently affect the degree of conservatism, avoiding judgment bias caused by a single confidence level dominating the process.

[0060] After obtaining two conservative components, the system directly selects the larger value between the first and second conservative components as the final conservative coefficient. Thus, as long as the prediction confidence at any time scale is low, the system treats the overall risk with the most conservative approach. The final conservative coefficient typically ranges from 0 to 1. When both confidence levels are high, the coefficient approaches 0, and the system hardly restricts mechanical movement; when either confidence level decreases significantly, the coefficient increases rapidly, forcing the system into a highly conservative state.

[0061] The final conservative coefficient is used for two purposes: First, the original target tracking angular velocity of the gimbal is multiplied by this coefficient and used as the maximum allowable angular velocity output to the motor, thus proportionally limiting the mechanical movement range and making the gimbal move slower and more cautiously. Simultaneously, the image cropping ratio of the electronic stabilization is dynamically adjusted to a preset multiple proportional to the final conservative coefficient; for example, the cropping ratio equals the base cropping plus the conservative coefficient multiplied by a multiplier. This proportional relationship ensures that the more the mechanical movement is restricted, the larger the compensation window of the electronic stabilization becomes, and the sum of the two always maintains the redundancy required for image stabilization, thus achieving true seamless complementarity.

[0062] Preferably, the present invention further includes: after determining that the energy recovery stabilization conditions are met and before the gimbal motor enters the regenerative braking state, performing a slight pre-braking of the current attitude of the gimbal for 0.1 to 0.4 seconds, so that the attitude of the gimbal is slightly deviated by a preset small angle in the predicted average motion direction of the group, and then formally entering the regenerative braking state and applying a reverse torque.

[0063] After the system determines that the energy recovery stabilization conditions are met, it does not directly put the motor into regenerative braking. Instead, it first inserts a weak pre-braking phase lasting 0.1 to 0.4 seconds. During this phase, through extremely slight active offset, the gimbal attitude is adjusted in advance to be slightly consistent with the future movement trend of the crowd. This changes the reverse torque generated by subsequent regenerative braking from "counteracting inertia" to "superimposing inertia with the trend," thus amplifying the torque many times over.

[0064] The specific method involves sending a brief pulse to the gimbal motor with an intensity only 10% to 30% of the normal tracking torque, lasting between 0.1 and 0.4 seconds. The direction of this pulse is strictly based on the average motion direction of the group calculated from the current long-term or short-term prediction results, causing the gimbal attitude to slightly shift in that direction by a preset small angle, typically 0.5 to 2 degrees, for a very short time. This shift is so small that it is almost imperceptible to the naked eye, yet it is sufficient to change the instantaneous inertial state of the gimbal.

[0065] After pre-braking, the system immediately puts the motor into regenerative braking mode. At this time, because the gimbal's attitude has been "pushed" in advance towards the average movement direction of the group, its inertial torque is consistent with the future movement trend of the crowd. When regenerative braking generates a reverse torque, this reverse torque no longer needs to overcome the inertia in the opposite direction, but is directly superimposed with the inertial torque in the same direction, forming a braking torque that is several times amplified. This superposition effect reduces the suppression speed of low-frequency jitter from the usual 1 to 2 seconds to less than 0.3 seconds, and the suppression amplitude is also significantly increased, achieving the ultimate image stability.

[0066] The method described in this application can be applied to various scenarios, including but not limited to: when filming after a concert or large performance, a handheld gimbal can be used to follow the main stage. When the crowd suddenly stops and waits for the exit, the system detects a high probability of stillness and triggers a regenerative braking mode, recovering energy equivalent to 50% of the normal power consumption within 10 seconds. Simultaneously, braking torque is used to suppress low-frequency jitter, resulting in a tripod-like stable image, completely solving the battery anxiety and jitter problems caused by crowds leaving the venue. In filming parades, rallies, or breaking news events, when the crowd suddenly stops or briefly freezes, this invention can instantly capture this high-confidence stillness window, accurately activating energy recovery stabilization. This extends the battery life of reporters during long handheld filming sessions while maintaining professional-grade stable image quality, avoiding image jumps or battery depletion caused by misjudgments in traditional gimbals.

[0067] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0068] According to another embodiment, a PTZ video group behavior prediction and stabilization system is provided. Figure 2 A schematic block diagram of a PTZ video crowd behavior prediction and stabilization system according to one embodiment is shown. Figure 2 As shown, the device 200 includes: The group behavior prediction unit 201 is configured to acquire the video stream captured in real time by the PTZ camera, perform short-term group behavior prediction and long-term group behavior prediction on the video stream, and obtain short-term prediction results, short-term prediction confidence, long-term prediction results and long-term prediction confidence.

[0069] The stationary probability determination unit 202 is configured to determine the probability that the group will remain stationary within a first preset time period in the future based on the long-term prediction result.

[0070] The energy recovery stabilization determination unit 203 is configured to directly determine that the energy recovery stabilization condition is met when the long-term prediction confidence is higher than the first threshold and the probability of the population remaining stationary is higher than the second threshold; otherwise, it further combines the short-term prediction confidence for a second determination, and only determines that the energy recovery stabilization condition is met when the short-term prediction confidence is higher than the third threshold.

[0071] The stabilization execution unit 204 is configured to, when the energy recovery stabilization conditions are met, control the gimbal motor to enter a regenerative braking state to recover electrical energy and use the reverse torque generated by the regenerative braking to mechanically stabilize the gimbal; when the energy recovery stabilization conditions are not met, limit the mechanical movement amplitude of the gimbal and increase the image cropping ratio of the electronic stabilization for compensation stabilization.

[0072] As an implementable approach, the group behavior prediction unit 201 can be configured such that the prediction duration for short-term group behavior prediction is 0.2 seconds to 1 second, and the short-term prediction result and short-term prediction confidence are obtained by fast trajectory extrapolation based on optical flow and pedestrian detection boxes.

[0073] As an implementable approach, the group behavior prediction unit 201 can be configured such that the prediction duration for long-term group behavior prediction is 4 to 15 seconds, and a lightweight time series model based on a multi-head attention mechanism is used to obtain the long-term prediction results and long-term prediction confidence.

[0074] As an implementable approach, the stationary probability determination unit 202, when determining the probability that the group will remain stationary within a first preset time period based on the long-term prediction results, can be configured to: convert the predicted trajectory of each individual in the long-term prediction results within the first preset time period into a corresponding set of predicted displacement vectors; cluster the set of predicted displacement vectors to obtain at least one motion consistency cluster; calculate the proportion of individuals in the largest motion consistency cluster among the total number of people and the average displacement within that cluster; and use the sum of the products of the proportion and a first weight, the reciprocal of the average displacement within the cluster and a second weight as the probability that the group will remain stationary.

[0075] As an implementable approach, the stabilization execution unit 204 can be configured to mechanically stabilize the gimbal using the reverse torque generated by regenerative braking as follows: after the gimbal motor enters the regenerative braking state, it collects the actual reverse torque value generated by the motor during the regenerative braking process in real time, and directly superimposes the reverse torque value as a feedforward compensation amount onto the torque output terminal of the original closed-loop attitude control loop of the gimbal; at the same time, it monitors the residual low-frequency jitter output by the gimbal angular velocity sensor, and when the direction of the residual low-frequency jitter is detected to be opposite to the current reverse torque direction, it instantaneously increases the regenerative braking current to actively amplify the reverse torque until the residual low-frequency jitter is suppressed to within a preset range.

[0076] As an implementable approach, the stabilization execution unit 204, when limiting the mechanical movement amplitude of the gimbal and increasing the image cropping ratio of electronic stabilization for compensation stabilization, can be configured as follows: First conservative component and second conservative component are respectively mapped based on the long-term prediction confidence and the short-term prediction confidence, with the conservative component monotonically increasing as the corresponding confidence decreases; the larger value between the first conservative component and the second conservative component is selected as the final conservative coefficient; the original target tracking angular velocity of the gimbal is multiplied by the final conservative coefficient to obtain the maximum allowable angular velocity actually output to the motor, while the image cropping ratio of electronic stabilization is dynamically adjusted to a preset multiple proportional to the final conservative coefficient.

[0077] As an implementable approach, the stabilization execution unit 204 can also be configured to: after determining that the energy recovery stabilization conditions are met and before the gimbal motor enters the regenerative braking state, perform a slight pre-braking of the current attitude of the gimbal for 0.1 to 0.4 seconds, so that the attitude of the gimbal is slightly deviated by a preset small angle in the predicted average motion direction of the group, and then formally enter the regenerative braking state and apply a reverse torque.

[0078] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. Components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0080] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0081] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0082] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0083] in, Figure 3 The architecture of an electronic device is illustrated, which may include a processor 310, a video display adapter 311, a disk drive 312, an input / output interface 313, a network interface 314, and a memory 320. The processor 310, video display adapter 311, disk drive 312, input / output interface 313, network interface 314, and memory 320 can communicate with each other via a communication bus 330.

[0084] The processor 310 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solution provided in this application.

[0085] The memory 320 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 320 can store the operating system 321 for controlling the operation of the electronic device 300, and the basic input / output system (BIOS) 322 for controlling the low-level operations of the electronic device 300. Additionally, it can store a web browser 323, a data storage management system 324, and a PTZ video crowd behavior prediction and stabilization system 325, etc. The aforementioned PTZ video crowd behavior prediction and stabilization system 325 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 320 and executed by the processor 310.

[0086] Input / output interface 313 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0087] Network interface 314 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0088] Bus 330 includes a pathway for transmitting information between various components of the device, such as processor 310, video display adapter 311, disk drive 312, input / output interface 313, network interface 314, and memory 320.

[0089] It should be noted that although the above-described device only shows the processor 310, video display adapter 311, disk drive 312, input / output interface 313, network interface 314, memory 320, bus 330, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0090] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0091] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for predicting and stabilizing group behavior in PTZ video, characterized in that, The method includes: The video stream captured in real time by the gimbal is acquired, and short-term and long-term group behavior predictions are performed on the video stream to obtain short-term prediction results, short-term prediction confidence, long-term prediction results, and long-term prediction confidence. Based on the long-term prediction results, determine the probability that the group will remain stationary within the first preset time period in the future; When the long-term prediction confidence is higher than the first threshold and the probability of the population remaining stationary is higher than the second threshold, it is directly determined that the energy recovery stabilization condition is met; otherwise, a second judgment is made in combination with the short-term prediction confidence, and the energy recovery stabilization condition is only determined to be met when the short-term prediction confidence is higher than the third threshold. When the energy recovery and stabilization conditions are met, the gimbal motor is controlled to enter regenerative braking state to recover electrical energy, and the reverse torque generated by regenerative braking is used to mechanically stabilize the gimbal; when the energy recovery and stabilization conditions are not met, the mechanical movement amplitude of the gimbal is limited and the image cropping ratio of electronic stabilization is increased for compensation stabilization.

2. The method according to claim 1, characterized in that, The prediction duration for the short-term group behavior prediction is 0.2 to 1 second. The short-term prediction result and short-term prediction confidence are obtained by fast trajectory extrapolation based on optical flow and pedestrian detection boxes.

3. The method according to claim 1, characterized in that, The prediction duration for the long-term group behavior prediction is 4 to 15 seconds. A lightweight time series model based on a multi-head attention mechanism is used to obtain the long-term prediction results and long-term prediction confidence.

4. The method according to claim 1, characterized in that, The determination of the probability that the group will remain stationary within a first preset time period based on the long-term prediction results includes: Convert the predicted trajectory of each person in the long-term prediction result within a first preset time period into a corresponding set of predicted displacement vectors; Clustering the predicted displacement vector set yields at least one motion-consistent cluster; The percentage of individuals within the largest uniformity of motion in the total population and the average displacement within that cluster are statistically analyzed. The sum of the product of the stated proportion, the first weight, the reciprocal of the average displacement within the cluster, and the second weight is taken as the probability that the group remains stationary.

5. The method according to claim 1, characterized in that, The method of using the reverse torque generated by regenerative braking to mechanically stabilize the gimbal includes: After the gimbal motor enters the regenerative braking state, the actual reverse torque value generated by the motor during the regenerative braking process is collected in real time, and the reverse torque value is directly superimposed on the torque output end of the original closed-loop attitude control loop of the gimbal as a feedforward compensation amount. Simultaneously monitor the residual low-frequency jitter output by the gimbal angular velocity sensor. When the direction of the residual low-frequency jitter is detected to be opposite to the direction of the current reverse torque, the regenerative braking current is instantaneously increased to actively amplify the reverse torque until the residual low-frequency jitter is suppressed to within the preset range.

6. The method according to claim 1, characterized in that, The method of limiting the mechanical movement range of the gimbal and increasing the image cropping ratio for electronic stabilization includes: The first conservative component and the second conservative component are respectively mapped based on the long-term prediction confidence and the short-term prediction confidence, and the conservative component increases monotonically as the corresponding confidence decreases. The larger value between the first conservative component and the second conservative component is selected as the final conservative coefficient. The original target tracking angular velocity of the gimbal is multiplied by the final conservative coefficient to obtain the maximum allowable angular velocity actually output to the motor. At the same time, the image cropping ratio of electronic stabilization is dynamically adjusted to a preset multiple that is proportional to the final conservative coefficient.

7. The method according to claim 1, characterized in that, The method further includes: after determining that the energy recovery and stabilization conditions are met and before the gimbal motor enters the regenerative braking state, performing a slight pre-braking of the current attitude of the gimbal for 0.1 to 0.4 seconds, so that the attitude of the gimbal is slightly offset by a preset small angle in the predicted average motion direction of the group, and then formally entering the regenerative braking state and applying a reverse torque.

8. A PTZ video group behavior prediction and stabilization system, characterized in that, The system includes: The group behavior prediction unit is configured to acquire the video stream captured in real time by the PTZ camera, perform short-term group behavior prediction and long-term group behavior prediction on the video stream, and obtain short-term prediction results, short-term prediction confidence, long-term prediction results and long-term prediction confidence. The stationary probability determination unit is configured to determine the probability that the group will remain stationary within a first preset time period in the future based on the long-term prediction result. The energy recovery stabilization determination unit is configured to directly determine that the energy recovery stabilization condition is met when the long-term prediction confidence is higher than a first threshold and the probability of the population remaining stationary is higher than a second threshold; otherwise, it further combines the short-term prediction confidence for a second determination, and only determines that the energy recovery stabilization condition is met when the short-term prediction confidence is higher than a third threshold. The stabilization execution unit is configured to, when the energy recovery stabilization conditions are met, control the gimbal motor to enter a regenerative braking state to recover electrical energy and use the reverse torque generated by regenerative braking to mechanically stabilize the gimbal; when the energy recovery stabilization conditions are not met, limit the mechanical movement amplitude of the gimbal and increase the image cropping ratio of electronic stabilization for compensation stabilization.

9. An electronic device, characterized in that, include: One or more processors; And a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 7.

Images