A separated handheld gimbal wireless networking system and data transmission method

Abstract

The embodiment of the application provides a kind of separated handheld gimbal wireless networking system and data transmission method, it is related to the technical field of separated handheld gimbal control technology.Its method includes: after receiving modal activation instruction, current modal is switched to target pre-control modal, and multiple target candidate set is determined according to the target environment image after processing;Get switching intention vector;Based on switching intention vector and the multiple target candidate set, switching score calculation is executed to determine new tracking target;According to new tracking target, target motion trajectory is planned, and target motion trajectory includes the action trajectory of the follow-up driving component from current posture to the posture corresponding to new tracking target.The application solves the problem of gimbal multi-target switching response delay, and reduces the response time of gimbal multi-target switching.

Description

Technical Field

[0001] This invention relates to the field of separate handheld gimbal control technology, and more specifically, to a separate handheld gimbal wireless networking system and data transmission method. Background Technology

[0002] As professional film and television shooting demands increasing flexibility from equipment, physically separating the remote control unit of the gimbal from the execution unit carrying the camera, and then controlling them via a wireless network, has become an important development direction for handheld gimbal systems.

[0003] While current wireless networking architectures have freed up operators' movement, in a wireless environment, the transmission of user operation commands inevitably involves link establishment, data packaging, over-the-air transmission, and decoding, resulting in communication latency. Furthermore, to achieve accurate tracking of specific targets in dynamic scenes (especially when switching between multiple potential targets), existing technologies typically employ the following approach: the operator issues a switching command (such as swiping on the screen), which is transmitted to the gimbal via the wireless network; upon receiving the command, the gimbal initiates complex visual processing algorithms to identify and locate the new target, and finally plans and executes the movement.

[0004] In this process, the wireless transmission of commands itself introduces a certain communication delay; secondly, the identification and locking of new targets requires significant computing resources, resulting in computational delays. The combination of these two delays leads to a significant time lag between the user's intention to switch and the gimbal's physical response. In applications requiring rapid switching of tracked objects (such as in sports events or multi-person conversations), this lag severely impacts the smoothness of operation and the professionalism of the camera. Summary of the Invention

[0005] This invention provides a wireless networking system and data transmission method for a separate handheld gimbal, which at least solves the problem of delayed multi-target switching response in related technologies.

[0006] According to an embodiment of the present invention, a wireless networking data transmission method for a detachable handheld gimbal is provided, applied to a detachable handheld gimbal. The detachable handheld gimbal includes a follow-shot drive component, a remote control component, and an image acquisition component. The remote control component is communicatively connected to the follow-shot drive component and the image acquisition component via a wireless communication network. The follow-shot drive component drives the image acquisition component to move to a target position. The method includes: Upon receiving a modal activation command, the current modality is switched to the target pre-control modality, and a multi-target candidate set is determined based on the processed target environment image; wherein, the environment image is acquired by the image acquisition device; the switching operation is performed after mapping the original touch command of the remote control device; the multi-target candidate set is obtained based on a deep learning target detection model; Obtain the switching intent vector; Based on the switching intent vector and the multi-target candidate set, a switching score is calculated to determine a new tracking target; Based on the new tracking target, the target motion trajectory is planned, which includes the motion trajectory of the follow-up driving component from the current posture to the corresponding posture of the new tracking target.

[0007] In one exemplary embodiment, after switching the current mode to the target pre-controlled mode, the method further includes: Based on the obstacle targets in the multi-target candidate set, an obstacle avoidance risk function is constructed; The motion trajectory is subjected to security verification; If the obstacle avoidance risk value of the motion trajectory exceeds the risk threshold, the motion trajectory is corrected to generate an obstacle avoidance motion trajectory.

[0008] In one exemplary embodiment, the method further includes: In touch mode, a remote control operation vector is obtained, which includes a synchronization timestamp; and the remote control operation vector corresponds to the operation action performed on the remote control device. The remote control operation vector is broadcast to the tracking drive component and the image acquisition component via the wireless communication network; The following drive component generates a first motion command based on the remote control operation vector and executes the first motion corresponding to the first motion command; The image acquisition device performs visual feature calculations on the acquired environmental image based on the remote control operation vector and the preset tracking area to obtain a correction vector; wherein, the correction vector includes the acquisition timestamp of the environmental image; The historical motion state of the first motion executed by the tracking drive component at the acquisition timestamp is fused with the correction vector to obtain a target drive command. The target drive command is used to correct the first motion to execute the second motion.

[0009] In an exemplary embodiment, fusing the historical motion state of the tracking drive component executing the first motion at the acquisition timestamp with the correction vector to obtain the target drive command includes: The angular deviation contained in the correction vector is used as the fusion vector; The fusion vector is fused with the historical state using a first filter to obtain the target driving command.

[0010] In one exemplary embodiment, the method further includes: In the visual gesture mode, the image acquisition device acquires image data and parses the image data to obtain the gesture data and position data of the person in the image; The image acquisition device sends the gesture data and the location data to the follow-up driving component; The follow-up driving component adjusts the first movement based on the type of the gesture data to perform a third movement that maintains the following of the person in the image.

[0011] In one exemplary embodiment, generating the first motion command based on the remote control operation vector includes: Based on a preset first mapping relationship, the screen pixel velocity contained in the remote control operation vector is converted into a target angular velocity; wherein, the target angular velocity is obtained based on a sliding ratio coefficient, which is determined according to the screen pixel velocity scalar, a specific direction component, and the total number of pixels at the resolution; Based on the target angular velocity, a first motion command is generated.

[0012] In one exemplary embodiment, the step of performing visual feature calculation on the environmental image to obtain a correction vector includes: Based on the remote control operation vector, the region of interest is updated, and the center coordinates of the visual target are determined; Based on the pixel deviation between the center coordinates and the optical center of the image, and the preset intrinsic parameter matrix of the image acquisition device, the pixel deviation is converted into an angle deviation; The correction vector is generated based on the angular deviation.

[0013] According to another embodiment of the present invention, a separate handheld gimbal wireless networking system and transmission method are provided, including a separate handheld gimbal, wherein the separate handheld gimbal further includes: Remote control unit, used to acquire original touch commands in touch mode; The image acquisition device is connected to the remote control device via a wireless communication network and is used to acquire environmental images; The tracking drive component communicates with the remote control unit via a wireless communication network. Upon receiving a modal activation command, the system switches the current modality to the target pre-control modality and determines a multi-target candidate set based on the processed target environment image. The switching operation is performed after mapping the original touch command of the remote control device. The multi-target candidate set is obtained based on a deep learning target detection model. A switching intent vector is acquired. Based on the switching intent vector and the multi-target candidate set, a new tracking target is determined. According to the new tracking target, a target motion trajectory is planned, including the motion trajectory of the tracking drive component from its current posture to the posture corresponding to the new tracking target.

[0014] In an exemplary embodiment, the remote control is further configured to obtain a remote control operation vector according to the original touch command, and broadcast the remote control operation vector through the wireless communication network; the remote control operation vector includes a synchronization timestamp, and the remote control operation vector corresponds to the operation action performed on the remote control; The tracking drive component is also used to generate a first motion command based on the remote control operation vector and the preset tracking area, and to execute the first motion according to the first motion command; The image acquisition device is also used to perform visual feature calculation on the environmental image based on the remote control operation vector and the preset tracking area to obtain a correction vector; wherein, the correction vector includes the acquisition timestamp of the environmental image; The tracking drive component will also fuse the historical motion state of the first motion executed at the acquisition timestamp with the correction vector to obtain a target drive command, and execute a second motion according to the target drive command; the target drive command is used to correct the first motion.

[0015] In one exemplary embodiment, it further includes: In the visual gesture mode, the image acquisition device acquires image data and parses the image data to obtain the gesture data and position data of the person in the image; The image acquisition device sends the gesture data and the location data to the follow-up driving component; The follow-up driving component adjusts the first movement based on the type of the gesture data to perform a third movement that maintains the following of the person in the image.

[0016] This invention constructs a multimodal interaction framework. After activating the pre-controlled modality, it uses a deep learning target detection model to pre-process the environmental image, actively identifying and establishing a multi-target candidate set containing multiple potential tracking objects. When the user issues a switching intention via remote control, the system no longer needs to perform time-consuming target detection and recognition processes. Instead, it quickly matches and scores the user's operation (i.e., the "switching intention vector") with the existing candidate set, thereby determining the new tracking target almost instantaneously. This effectively shortens the decision-making time from user input to system determination of the new target, thus solving the problem of gimbal communication and computation lag under multiple tracking targets and achieving a real-time and smooth response to changes in the tracking target. Attached Figure Description

[0017] Figure 1 This is the structure of the detachable handheld gimbal in this embodiment of the invention. Figure 1 ; Figure 2 This is the structure of the detachable handheld gimbal in this embodiment of the invention. Figure 2 ; Figure 3 This is a flowchart of a wireless networking data transmission method for a separate handheld gimbal according to an embodiment of the present invention; Figure 4 This is a flowchart of step S300 in an embodiment of the present invention; Figure 5 This is a flowchart of the third interaction mode in an embodiment of the present invention.

[0018] In the diagram, 1 is the handheld component; 21 is the stabilizer; 221 is the first adapter arm; 222 is the lifting control arm; 223 is the lifting control platform; 311 is the first follow-up camera driver; 312 is the second follow-up camera driver; 313 is the second adapter arm; 321 is the third follow-up camera driver; 322 is the third adapter arm; 33 is the limiting component; 34 is the image acquisition component; 35 is the mounting platform; and 4 is the remote control component. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0020] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0021] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0022] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0023] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0024] This embodiment provides a method for wireless networking data transmission using a separate handheld gimbal. This method is applied to a wireless networking system for separate handheld gimbals, such as... Figure 1-2 As shown, the system includes a detachable handheld gimbal, which includes a handheld component 1, a jug-lifting motion component, a remote control component 4, and a follow-up shooting component.

[0025] The handheld component 1 has a non-slip texture for easy gripping; the kettle-lifting motion assembly includes a stabilizer 21 hinged to the end of the handheld component 1 and a kettle-lifting assembly for saving effort and reducing burden and for low-angle shooting. The kettle-lifting assembly includes a first adapter arm 221 fixed to the top of the stabilizer 21, an upward lifting operation arm 222 hinged to the end of the first adapter arm 221 away from the stabilizer 21, and an upward lifting operation platform 223 fixed to the end of the upward lifting operation arm 222 away from the first adapter arm 221; the kettle-lifting assembly, together with the handheld component 1, can adapt to shooting at different angles.

[0026] The remote control unit 4 is fixed to one side of the stabilizer 21 for convenient remote control operation. The remote control unit 4 can be a touchscreen, operation buttons, or a combination of both; no limitation is made here. The remote control unit 4 integrates a touch controller and a microcontroller unit (MCU). The touch controller captures the original touch point coordinates and timestamp sequence (i.e., the original touch commands) at a sampling rate of 120Hz. The MCU analyzes the touch point coordinates and timestamp sequence to extract key motion features such as motion direction and instantaneous speed. When a specific trigger logic is met (e.g., the speed continuously exceeds a threshold), these features, along with a global synchronization timestamp, are encoded into a structured data packet, which is the remote control operation vector. Subsequently, the remote control unit 4 broadcasts the remote control operation vector via its built-in wireless communication module using the User Datagram Protocol (UDP) multicast method.

[0027] The tracking component includes an image acquisition unit 34 for image acquisition and analysis, and a tracking drive component (i.e., a three-axis gimbal main body) for driving the image acquisition unit 34 to the target position. The tracking drive component has a main control unit integrated within the image acquisition unit 34. The image acquisition unit 34, the main control unit, and the remote control unit 4 are interconnected via a wireless communication network based on Wi-Fi or Bluetooth technology. The tracking drive component includes a first tracking drive 311 (i.e., a yaw axis) located at the end of the stabilizer 21 away from the handheld component 1, and a second tracking drive 311 connected to the first tracking drive via a second adapter arm 313. 12 (i.e., pitch axis), a third follow-up drive 321 (i.e., roll axis) connected to the second follow-up drive 312 via a third adapter arm 322, and a limiting member 33 connected to the third follow-up drive 321 via a mounting platform 35. The limiting member 33 is used to clamp mobile devices such as mobile phones that can perform image acquisition. An image acquisition device 34 is fixed at the end of the mounting platform 35 away from the third follow-up drive 321. The image acquisition device 34 includes a camera. When the follow-up drive assembly receives an action command, it drives the adapter arm to move, thereby moving the mobile device clamped by the limiting member 33 and / or the image acquisition device 34 to the target position for image acquisition.

[0028] Image acquisition unit 34 can be a smartphone with an integrated camera or a dedicated camera integrating a SoC system and an image sensor. Image acquisition unit 34 is primarily used for visual analysis and generating correction information to correct motion trajectories. After receiving a remote control operation vector, image acquisition unit 34 predictively adjusts its region of interest (ROI) in the next frame image based on the direction and speed indicated by the vector, thereby narrowing the search range and improving target tracking success rate and computational efficiency in high-speed motion or brief occlusion situations. After locking onto the visual target, image acquisition unit 34 processes each frame in the video stream to calculate the target's center position in the current image coordinate system. Subsequently, based on pre-calibrated camera intrinsic parameters (such as focal length and optical center), the pixel deviation of the target position is converted into angular deviations of the yaw and pitch axes in the gimbal coordinate system. This angular deviation, along with the global synchronization timestamp of the image frame and a confidence score characterizing tracking quality, is then encapsulated into a correction vector. Finally, image acquisition unit 34 sends this correction vector to the tracking driver component via UDP unicast.

[0029] To reduce the perceived lag during tracking, the follow-up drive component executes a two-stage motion upon receiving the remote control operation vector. Specifically, in the first stage, the follow-up drive component maps the screen pixel velocity to the target angular velocity based on a preset mapping relationship and generates torque commands to drive the first follow-up drive 311, the second follow-up drive 312, and the third follow-up drive 321. This stage ensures that the gimbal's response occurs almost instantly upon the user's finger swipe, significantly reducing perceived latency. Simultaneously with the first stage, the state estimator within the follow-up drive component (e.g., an extended Kalman filter, i.e., the first filter) fuses IMU readings and drive encoder data at a frequency of 1000Hz, continuously estimating and recording the gimbal's attitude (angle, angular velocity) at each moment, thus forming a timestamped state history trajectory. Then, in the second stage, when the follow-up drive component receives a correction vector, it first indexes and extracts the corresponding predicted gimbal state from the state history trajectory based on the historical timestamp in the correction vector. Then, the angle deviation contained in the correction vector is used as an external measurement and fused with the historical state to obtain the corrected state; then, the target motion command is generated based on the corrected state to guide the gimbal to perform the second motion.

[0030] Specifically, refer to Figure 3 ,include: S000: Environment Initialization When the tracking component is started for the first time or when the mounting is changed, a preset excitation signal is injected into its motor and its response is recorded. Then, the motion parameters of the tracking drive component are identified online through the system's identification algorithm to build a dynamic model. Of course, the model can also be built directly by inputting parameters, which is not limited here.

[0031] When the system performs initialization operations, an initialization reset level is generated; similarly, when a load change occurs, the corresponding load device descriptor will be changed; when the main control unit receives the initialization reset level or detects a change in the device descriptor, the main control unit outputs a current torque command to adjust the motor torque to 15% of the rated torque, thereby preventing mechanical limit collisions.

[0032] Simultaneously, during the excitation signal injection, the main control unit synchronously acquires orthogonal angular acceleration and angular velocity data sequences of the gimbal's three axes (i.e., the second adapter arm, the third adapter arm, and the mounting platform). These data sequences are typically acquired using an integrated six-axis inertial measurement unit and a magnetic encoder at a sampling rate of 8000Hz. The main control unit then defines the applied current-torque command sequence as the input matrix U and the acquired angular velocity response sequence as the output matrix Y. Subsequently, the system iteratively minimizes the sum of squared prediction errors between the actual angular velocity sequence and the angular velocity sequence calculated from the current model parameters. From this, the system calculates the gimbal's moment of inertia parameters and mixed friction coefficient parameters such as viscous damping coefficient in each independent orthogonal axis. For example, the moment of inertia J of the first tracking drive is calculated. yaw It is 0.015 kg·m 2 Viscous damping coefficient D yaw It is 0.002 N·m·s / rad, and so on.

[0033] The main control unit then executes the following interaction modality: Interaction Mode 1: Touch Prediction and Fusion Correction Based on Remote Control Operation Vectors In this mode, the remote-controlled component broadcasts vector data to drive the follow-up motion drive assembly in a feedforward motion. Specifically: S100: Obtain the remote control operation vector corresponding to the touch operation on the remote control device.

[0034] When a user operates the touchscreen, its internal electrostatic capacitive field changes as the operation progresses. At this time, the capacitive touch array integrated circuit of remote control 4 submits a hardware interrupt to the main processor. Subsequently, remote control 4 reads a set of data frames at a clock frequency of 120Hz. This data frame contains the current pixel coordinates of the screen touch point and a timestamp. Each newly read data frame is then structured into a data sequence containing a horizontal coordinate field, a vertical coordinate field, and a timestamp field, thereby obtaining the original touch command, which is then written into the address block based on modulo operations.

[0035] For example, assume that at system time t sys At 150000.008 seconds, the following data was obtained: horizontal pixel x = 960 pixels, vertical pixel y = 540 pixels, and then this data was stored.

[0036] Subsequently, the difference in screen coordinates between two adjacent sampling points is extracted by the feature extraction unit and divided by the time difference of their respective timestamp labels to obtain a two-dimensional instantaneous velocity vector; then, the Euclidean norm of this two-dimensional velocity vector is calculated to obtain the instantaneous velocity scalar V. inst .

[0037] The preset trigger condition is a dual-constraint logic gate, which continuously monitors the instantaneous velocity scalar V. inst If and only if the instantaneous velocity scalar V inst Exceeding the preset motion speed threshold V threshold Furthermore, the duration of this overshoot state on the timeline exceeds the anti-shake filtering time window T. window When the time condition is met, the trigger logic is true.

[0038] For example, assuming the system time is 150000.23000000, the speed threshold V threshold The shutter speed is 50.0 pixels per second, and the image stabilization filtering time window T is... window The time is 15.0 milliseconds. If the trigger logic is true, the feature extraction unit calculates the arithmetic mean of all discrete instantaneous velocity scalars within the window period, and uses this arithmetic mean as the feature velocity scalar V. scalar The coordinates of the beginning and end of the window period are normalized to obtain a two-dimensional direction vector. After encoding, the remote control operation vector can be obtained as follows: {Type:0x02,DirX:0.866,DirY:0.500,Vel:450.5,Timestamp:15000023000000}, and so on.

[0039] S200: The remote control operation vector is broadcast to the tracking drive component and the image acquisition component via the wireless communication network.

[0040] The remote control unit 4 broadcasts the remote control operation vector as a UDP protocol payload. Since UDP has low overhead and high propagation speed, it is suitable for information transmission between PTZ units. For example, the target multicast address is set to 224.0.1.25 and the port to 8888. This ensures that any device in the network that joins this multicast group can receive data packets destined for that address and port. Because the propagation speed of wireless signals in space is close to the speed of light, and multicast is a highly efficient one-to-many distribution at the network layer, it can be assumed that the tracking drive component and the image acquisition unit 34 receive the data packet almost simultaneously.

[0041] S300: The tracking drive component receives the remote control operation vector, generates a first motion command based on the preset tracking area and the remote control operation vector, and executes the first motion according to the first motion command.

[0042] Specifically, such as Figure 4 As shown, it includes the following sub-steps: S310: Based on a preset cross-domain mapping relationship, convert the screen pixel velocity contained in the remote control operation vector into the target angular velocity.

[0043] The cross-domain mapping relationship consists of a pair of algebraic proportional equations, constructed based on orthogonal decoupling axes, used to map geometric sliding events in discrete screen space to kinematic constraints driving rigid body rotation in the three-dimensional physical world. For example, for the target angular velocity of the yaw axis, the master control unit extracts the screen pixel velocity scalar V from the remote control operation vector. scalar and direction vector component D irx and Diry The cross-domain mapping then applies the screen pixel velocity scalar V. scalar The product of the components in a specific direction is used as the numerator, and the total number of pixels R of the screen of the remote control 4 in the corresponding dimension is calculated. x As the denominator, the sliding proportion coefficient is obtained. : ; This sliding percentage coefficient Subsequently, the maximum field of view (FOV) of the camera lens mounted on the image acquisition unit 34 along the corresponding axis is... h Multiply by the given gain K, and finally multiply by the pre-calibrated operating sensitivity constant. s This allows the movement of pixels on the screen to be converted into the required rotational angular velocity of the gimbal (i.e., the corresponding target angular velocity).

[0044] For example, let the screen horizontal resolution width W screen 1920 pixels, camera lens horizontal field of view (FOV) HThe gain K is 84.0°, and the operating sensitivity constant is 84.0°. s Set it to 1.2. When V is extracted... scalar =450.5 pixels / second and D irx When the value is 0.866, the microcontroller's floating-point arithmetic unit performs the operation. The final calculated yaw rate of the physical target is Degrees per second, and so on.

[0045] It's important to note that different smartphones or remote controls have drastically different screen sizes and pixel densities (e.g., 1080P, 2K, 4K screens). Using traditional pixel-level sliding speed to control the gimbal would result in excessively fast rotation on low-resolution screens and excessively slow rotation on high-resolution screens when sliding the same distance. Therefore, the absolute pixel speed (i.e., the screen pixel speed scalar V) will be used here. scalar The product of the component in a specific direction and the total screen resolution (roughly divided by the total screen resolution) converts the hardware specifications of a specific device into a relative percentage, that is, a fraction of the screen width swiped per second. Then, the swipe percentage coefficient is calculated. With optical maximum field of view (FOV) h Multiplication can effectively reflect the intuitiveness of human-computer interaction and the smoothness of operation, because at this time, the user's finger swipes across X% of the screen width on the control terminal, and the gimbal precisely rotates the camera's FOV by X in the physical world.

[0046] S320: Generates the first motion command based on the target angular velocity and the preset dynamic model.

[0047] After obtaining the target angular velocity sequence, the required target angular acceleration is approximately calculated using the Euler difference method. ,in The control cycle period of the bottom torque loop (e.g., 0.001 seconds).

[0048] Then the target angle acceleration measurement was performed. Substituting the moment of inertia J yaw and viscous damping coefficient D yaw The inverse equation formed Therefore, the feedforward electromagnetic torque that must be instantaneously applied to the rotor shaft is obtained. .

[0049] For example, suppose the current Substituting into the equation yields... Then, a feedforward torque command is generated based on this torque amount, causing the gimbal to produce a predictive following motion (i.e., the first motion).

[0050] S400: The image acquisition unit receives the remote control operation vector and performs visual feature calculation on the environmental image based on the remote control operation vector, thereby generating a correction vector for correcting predictive motion.

[0051] Specifically, it includes the following sub-steps: S410: Based on the remote control operation vector or the target selected through the remote control screen, update the region of interest of the visual tracking algorithm to lock onto the specific visual target.

[0052] The backbone process of image acquisition unit 34 captures data packets of remote control operation vectors on the corresponding network port. In this stage, the system supports two visual target locking methods.

[0053] In the first approach, the image acquisition unit 34 parses the broadcast remote control operation vector, maps the user's sliding trajectory intention on the screen of the remote control unit 4 to the coordinate system of the current video frame, and uses the trajectory to infer the endpoint as the center to forcibly cover and generate a new region of interest (ROI) (i.e., tracking region). The new ROI is generally smaller in size and has more concentrated pixels. In this way, when the next frame of video data is received, visual tracking algorithms such as SiamRPN no longer need to search the entire screen, but only within the ROI, which effectively reduces the amount of computation and improves the robustness of tracking.

[0054] In the second method, the user can directly tap or select a specific person or target in the image by touching the screen. The pixel coordinate data generated by this operation is sent to the image acquisition unit 34 through an independent high-priority control command.

[0055] After receiving the updated region of interest or target coordinates, the image acquisition unit 34 instantiates the high-frequency feature activation map in the region and saves it as a brand-new template branch kernel through the SiamRPN framework, thereby achieving hard locking of the target.

[0056] S420: Outputs the center coordinates of the visual target in the image through a visual tracking algorithm.

[0057] After locking onto and acquiring the next frame of the search image, the image acquisition unit 34 extracts the feature map of the search image and performs a deep cross-correlation calculation with the newly instantiated template branch kernel in S410. Through the RPN (Region Proposal Network) subnetwork within the SiamRPN framework, the cross-correlation results are fed into two independent fully convolutional branches: the classification branch outputs a response score heatmap of the target's presence probability based on the similarity between the template and the search features; the regression branch outputs the target's position and size offset relative to the preset anchor box, i.e., the regression branch parameters. Subsequently, the image acquisition unit 34 searches for the point of maximum probability activation in the response score heatmap and, combined with the regression branch parameters, calculates the pixel center coordinates of the tracked visual target in the current full-frame two-dimensional array. .

[0058] S430: Based on the pixel deviation between the center coordinates and the image optical center, and the camera's preset intrinsic parameter matrix, the pixel deviation is converted into an angular deviation to generate a correction vector.

[0059] The geometric calibration procedure within the image acquisition unit 34 will adjust the pixel center coordinates (C... x C y (and the reference coordinates of the geometric center of the optical sensor) Perform interpolation to separate pixel deviations. and : Subsequently, the camera's factory-calibrated intrinsic parameter matrix data (equivalent physical focal length) is retrieved. and Then, the trigonometric inverse tangent function is performed to obtain the corresponding angle deviation: ; In the formula, It's a yaw angle deviation. These two deviations, representing the pitch angle deviation, are used to indicate the difference between the actual target position and the gimbal's movement position.

[0060] For example, calculated Pixels Pixels, substituting into the trigonometric inverse tangent function, yield And so on. The angle deviation, along with the timestamp and confidence score, is then packaged into a correction vector, Correction_Vec, and sent back to the tracking driver component.

[0061] S500: The tracking drive component receives the correction vector and fuses the historical motion state of the first motion executed by the tracking drive component at the acquisition timestamp with the correction vector to generate a target drive command, thereby correcting the first motion.

[0062] This correction process aims to eliminate asynchronous temporal misalignment caused by complex image processing. It specifically includes two sub-steps: vector fusion and motion correction. S501: The historical motion state of the first motion executed by the tracking drive component at the acquisition timestamp is fused with the correction vector to obtain the fused correction state.

[0063] During the first phase of motion, the extended Kalman filter of the tracking drive component determines the gimbal attitude at the current moment based on the state of the previous moment and the current motor control input. This attitude includes yaw and pitch angles, angular velocities, etc. Subsequently, the current gimbal attitude, along with its covariance matrix and the current timestamp, constitutes a state history sequence.

[0064] At some future moment (e.g., t = 1.110 seconds, considering a 15-millisecond wireless transmission delay), the tracking driver component receives the correction vector. The tracking driver component then first reads the acquisition timestamp from the vector, i.e., t. m =1.050 seconds. Then, perform a time-series backtracking operation: that is, in the state history sequence, locate and extract the historical state X stored at timestamp 1.050 seconds through binary search or a timestamp-based hash index. pred (t m ) and covariance matrix P pred (t m This process involves generating a historical motion state, which is then fused with a correction vector using a Kalman filter framework. The correction vector includes angular deviations. This is equivalent to the residual In in Kalman filtering. Therefore, there is no need to perform the traditional calculation of subtracting the predicted measurement from the measured value.

[0065] Subsequently, a state update is performed: first, the Kalman gain K is calculated, and then based on... To update the state vector, and according to To update the covariance matrix, where X p For the corrected historical state, P p Here, I is the corrected covariance matrix, and H is the observation matrix; This represents the amount of correction for uncertainty in this measurement. This represents the proportion of uncertainty after the correction. Updated state X p (t m ), that is, being t m The historical state corrected by the visual truth value at any given moment, i.e. the fusion correction state.

[0066] For example, at t=1.110 seconds, the tracking drive component receives the correction vector {frame_timestamp:1.050s, error_yaw: 0.197, ...}. It immediately finds the state record at t=1.050 seconds in the history buffer. Assuming the predicted yaw angle at that time was 3.0 degrees, the gain K=0.8, and the residual was 0.197 degrees, substituting these values, we can obtain... Degrees. This 3.1576 degrees represents the fusion correction state at t=1.050 seconds.

[0067] S502, the follow-up drive component generates a corrected second motion based on the fusion correction state.

[0068] At the historical moment t m Fusion correction state X p (t m After that, it is also necessary to determine from t m to t curr After this period of motor control, the follow-up drive component is at t curr The state at any given moment.

[0069] Here, X p (t m Starting from t, then extract the values ​​from t. m +1 to t curr Within a given time period, the system actually applies a sequence of control commands to the motor: {u(t_m+1), ..., u(t_curr)}. Then, the prediction equations using Kalman filtering are... Re-analyze the motion trajectory of the gimbal to obtain t curr Optimal state estimate X at time t op (t curr This optimal state not only includes the latest IMU and motor information but also eliminates accumulated errors caused by delays. Finally, the optimal state X is... op (t curr The target setpoint is used as the motor's PID (proportional-integral-derivative) setting point. The controller then compares the current actual state with the optimal target state, generates a target drive command, and instructs the first follow-up drive 311, the second follow-up drive 312, and the third follow-up drive 321 to execute a second motion, causing the gimbal to face the visually tracked target.

[0070] For example, starting from the fusion correction state at t=1.050 seconds (yaw angle 3.1576 degrees), the system utilizes 60 control instructions between t=1.051 seconds and t=1.110 seconds to perform high-speed recalculation in memory. Assume the recalculated optimal target yaw angle at t=1.110 seconds is 30.0 degrees, while the actual yaw angle caused by the first predictive motion is 32.0 degrees. Then, the PID controller will generate a negative correction torque, causing the gimbal to decelerate from 32.0 degrees to 30.0 degrees, thereby achieving alignment with the visual target.

[0071] Interaction Mode 2: Autonomous Following Control Based on Visual Gestures To further adapt to complex operational scenarios such as outdoor inspections where individuals operate alone and their hands are restricted, this embodiment also constructs a visual gesture autonomous control link based on wireless networking and operating in parallel with touch mode. Specifically: S600: Image acquisition unit 34 acquires and analyzes image data to calculate the gesture data and position data of the person in the image.

[0072] Specifically, it includes the following sub-steps: S601: Construct a dual-track visual data stream protocol.

[0073] The image acquisition unit 34 controls the camera or image sensor to acquire raw images at a stable frame rate (e.g., 60 frames per second). The acquired high-definition video data is sent to a hardware video encoding engine (e.g., an H.265 encoder) for high-ratio compression to generate a basic image data stream (corresponding to data a in the parameter settings). Simultaneously, the wireless image transmission communication subsystem built into the image acquisition unit 34, based on a custom low-latency WIFI / OFDM communication protocol stack, encapsulates the video stream data a and continuously transmits it transparently to the remote control unit 4. After receiving data a, the remote control unit 4 calls its own hardware decoder to reconstruct the image and renders and maps the real-time monitoring image onto the high-brightness display screen of the remote control unit 4, thereby enabling the operator to obtain a first-person perspective of the image acquired by the image acquisition unit in real time.

[0074] Step S602: Parallel gesture feature cascade detection.

[0075] While pushing the video stream to the remote control unit 4, the image acquisition unit 34 allocates raw image frames in its local memory pool. Subsequently, the Tensor Processing Unit (TPU) inside the image acquisition unit 34 loads a joint detection cascade model. This joint detection cascade model is a multi-task deep learning model composed of a lightweight human pose estimation network (such as Lite-HRNet or YOLO-Pose) for first-level coarse localization and a hand target detection network (such as lightweight SSD or YOLO-Fastest) for second-level fine classification. This model is pre-trained on a large-scale open-source dataset containing human keypoints and gesture classification labels, and then deployed to the TPU after pruning and INT8 quantization compression. The model first identifies potential subjects (people) within the frame through a human skeleton network and extracts the pixel coordinates of these subjects, using this as positional data representing the subject's displacement. Meanwhile, the region extraction algorithm used in the front-end and back-end networks locates the spatial regions of the human wrist and palm based on the skeletal joints. The specific execution logic of this algorithm is as follows: First, extract the two-dimensional coordinates of the elbow and wrist joints output by the human skeletal network; calculate the direction vector from the elbow to the wrist; then, along this direction vector, extend outwards by a predetermined proportion (e.g., 20%-30% of the distance between the two joints) from the wrist joint to estimate the coordinates of the potential palm center point; finally, using the palm center point as a reference, crop a rectangular bounding box according to a predetermined pixel scale to extract the spatial region image patch where the wrist and palm are located. Subsequently, this spatial region image patch is transmitted to the back-end gesture classification convolutional neural network, which has multiple manually defined semantic label classes. When the Softmax confidence probability of a specific gesture label (such as the "OK" gesture representing confirmation of activation, or the "STOP" gesture with a full palm open representing emergency stop) output by the network inference layer exceeds a preset judgment threshold (e.g., 0.85) for three consecutive frames, the image acquisition unit 34 will structure and package the classification result to generate specific gesture data.

[0076] S700: The image acquisition unit 34 sends the gesture data and the location data to the follow-up driving component.

[0077] The image acquisition unit 34 encapsulates the calculated discrete gesture data (as state machine control signals) and continuous position data (as position closed-loop feedback reference points) into high-priority instruction packets according to a preset frame format and sends them to the main controller of the tracking drive component.

[0078] S800: The follow-shot drive component controls the start and stop of the first movement based on the type of gesture data, and adjusts the gimbal attitude in real time based on position data to perform a third movement to keep following the subject.

[0079] The state machine of the follow-up driver component is typically in "touch-driven mode" or "idle hover mode" by default. When the follow-up driver component parses valid gesture data, it immediately executes the state arbitration mechanism: If the parsed gesture data is of the "OK" gesture type, the state machine immediately triggers a state transition, entering the "visual autonomous target following mode". In this mode, the main control loop of the follow-up drive component suspends or reduces the weight of the touch prediction command from the remote control 4. The follow-up drive component then uses the continuously received subject position data (including pixel coordinate offset relative to the center of the screen) as an error source and sends it to the internally preset PID follow-up controller. Specifically, the system calculates the horizontal pixel difference (X-axis error) and vertical pixel difference (Y-axis error) between the target center pixel coordinates and the current physical optical center reference point of the image acquisition device. Then, the above errors are input into the internally preset PID follow-up controller. The controller calculates the pixel error through PID gain and converts it into the yaw axis and pitch axis compensation angular velocity to compensate for the target's motion trajectory. Subsequently, it drives the follow-up drive component to always align its optical axis with and lock onto the subject that issued the gesture command, realizing fully automatic composition tracking.

[0080] If, during the autonomous following process, the image acquisition unit 34 further analyzes the subject's "STOP" gesture and sends it to the follow-up driving component. Upon receiving this termination type data, the state machine of the follow-up driving component immediately blocks the output of the follow-up PID controller and simultaneously controls the rotors of the three-phase motors in the three drives to decelerate to a standstill through a smooth decay braking algorithm. Specifically, the smooth decay braking algorithm works as follows: the system records the instantaneous angular velocity of the current three-axis motors at the moment of receiving the termination command, and calls a preset exponential decay function or S-shaped speed planning curve. Within a preset braking buffer time window (e.g., 0.3-0.5 seconds), it generates a continuously decreasing angular velocity feedforward command, causing the motor rotors to decelerate smoothly and without significant oscillation to zero, thereby avoiding mechanical stress impact and video image ghosting caused by sudden braking of the gimbal. Subsequently, the system returns to the "idle hovering mode" and waits for the next gesture wake-up or touch control takeover by the remote control unit 4.

[0081] Interaction Mode 3: Intelligent Target Switching and Obstacle Avoidance Pre-control To adapt to the need for rapid viewpoint switching or proactive avoidance of potential line-of-sight obstructions in complex dynamic scenes, this embodiment maps the original touch commands input from the remote control to additional context-aware intent commands, thereby achieving adaptive control of the gimbal without increasing the operator's workload. Specifically, as shown... Figure 5 As shown, it includes the following sub-steps: S900: Activate intelligent target switching and obstacle avoidance pre-control mode To accommodate rapid switching and obstacle avoidance, a preset trigger mechanism is used to execute mode switching. Specifically, this is achieved through the information interaction interface of remote control unit 4, whose built-in processor continuously monitors and parses input events. When the processor detects a specific input event, the system switches from the current mode to interactive mode three.

[0082] Among them, a specific input event can be a specific multi-touch gesture. In order to clearly distinguish it from the single-point or double-point swipe and drag gestures used to directly control the gimbal movement and to avoid confusion of the operation intention, a specific input event can be a three-finger simultaneous light touch action on the touch screen of the remote control 4 for a specific duration.

[0083] To this end, the remote control 4 samples the state of the touchscreen at a frequency of 120 Hz. When it simultaneously detects three touch points with coordinate position changes less than the small displacement threshold δ within at least k consecutive sampling periods (e.g., k=3, corresponding to 25 ms), and the overall existence time T of these three touch points satisfies T min <T<T max (Assume T) min =50"ms", T max When the time interval is 500ms, the processor recognizes this event sequence as a valid modal activation instruction. The value of the displacement threshold δ is determined based on the physical pixel density of the touchscreen to ensure its universality. In one example, this threshold can be set to 0.5% of the screen diagonal length. For a 6-inch (152.4mm) screen with a resolution of 1920×1080 pixels, the diagonal pixel count is approximately 2202 pixels, so δ can be set to 11 pixels.

[0084] Upon receiving the modal activation command, the system state machine transitions to the target pre-control mode (i.e., interactive mode three) and sends an activation signal to the image acquisition unit 34.

[0085] For example, suppose an operator performs a light touch operation on the touchscreen using three fingers at t0 = 150000.800"s. The touch event processing unit of remote control 4 records three sets of touch point coordinates at three sampling times: t1 = 150000.825"s, t2 = 150000.833"s, and t3 = 150000.841"s. P1={(100,200),(400,210),(250,500)} P2={(101,200),(400,211),(251,502)} P3={(100,201),(401,210),(250,501)}, The processor calculates the displacement of the touch point corresponding to the same finger at any two adjacent time points. For example, the displacement of the first finger from t1 to t2 is... Pixels smaller than the displacement threshold Pixels. Meanwhile, finger touch... The time ends, and the total duration T = 0.950 - 0.800 = 150 ms, which falls within the preset time. Within the specified range. Therefore, the system determines this to be a valid three-finger touch gesture, immediately updates the ControlMode flag within the system from 0x01 (direct touch mode) to 0x03 (target pre-control mode), and generates a control information packet. It is broadcast to the image acquisition device 34 via a wireless link.

[0086] S1000: Scene Understanding and Multi-Objective Candidate Set Generation After the target pre-control mode is activated, in addition to executing the main computation path used in the aforementioned embodiment to perform visual feature calculation on the currently tracked target to generate a correction vector, the image acquisition unit 34 also performs real-time analysis on the acquired complete environmental image to identify and locate all potential objects in the image. These objects include (but are not limited to) other moving people, vehicles, static obstacles (such as trees, building pillars), etc., and generate a structured descriptor for each identified object, thereby forming a dynamically updated multi-target candidate set.

[0087] This analysis process is implemented using a deep learning object detection model; specifically, after acquiring each frame of original image data, the image processor of the image acquisition unit 34 first generates a copy of that frame of data; subsequently, the main computing path uses a high-resolution version of this copy (e.g., The first computational path performs precise target tracking on the region of interest (ROI) of a pixel, while the second computational path downsamples this copy to a lower resolution (e.g., pixel). The pixels are normalized, and the result is then fed as an input tensor into a deep learning object detection model. This deep learning object detection model uses a pruned and quantized optimized YOLO architecture, which can be decomposed into three main functional modules: the backbone network, the neck network, and the head network. These three modules then perform the following operations: S1110: Multi-scale feature extraction via backbone network The backbone network extracts visual features at different levels from the input image tensor, as defined below: Input tensors: The shape of the tensors input into this backbone network is... The dimensions correspond to the batch size, the number of RGB color channels, and the height and width of the image (in pixels).

[0088] Core Processing and Parameterization: The backbone network adopts the structure of CSPDarknet or EfficientNet. Taking CSPDarknet as an example, it processes the input tensor through a series of convolutional blocks (ConvBlock) and cross-stage local connectivity blocks (CSP). Each convolutional block contains a convolutional kernel of size [size missing]. The convolutional layer consists of a batch normalization layer and a SiLU activation function. The CSP block divides the feature map into two parts along the channel dimension. One part is directly connected by a short circuit, while the other part is processed through multiple convolutional blocks. Finally, the two parts are concatenated, thereby reducing the computational cost while maintaining the receptive field and feature extraction capabilities.

[0089] Output Tensors: The backbone network typically outputs three feature maps at different scales to detect targets of varying sizes. For example, the shapes of its three output feature maps might be... (Large scale, used to detect small targets) (mesoscale) and (Small scale, used to detect large targets).

[0090] S1120: Feature fusion via neck network The neck network fuses feature maps from different scales from the backbone network to enhance the model's contextual awareness of the target. It is defined as follows: Input tensor: Its input consists of three feature maps output by the backbone network.

[0091] Core Processing and Parameterization: The neck network can employ a combined structure of Feature Pyramid Network (FPN) and Path Aggregation Network (PANet). FPN uses a top-down path to upsample and pass high-level semantic features (from small-scale feature maps) to lower layers, fusing them with the localization details of the lower layers. PANet adds a bottom-up path to FPN, further passing strong localization features from the lower layers to the higher layers. This process involves numerous convolutions, upsampling (such as nearest neighbor interpolation), and concatenation operations.

[0092] Output tensors: The neck network also outputs three fully fused feature maps, with the same size as the inputs, but containing richer semantic and spatial information, and with the following shape. , and .

[0093] S1130: Target localization and classification via a detection head. The detection head performs the final prediction on the fused feature map, and its relevant definitions are as follows: Input tensor: Its input consists of three feature maps output by the neck network.

[0094] Core processing and parameterization: The detection head typically consists of multiple The system consists of convolutional layers. For each feature map at each scale, the detector head predicts several bounding boxes for each grid cell. The predicted information for each bounding box includes its center point coordinates (b... x ,b y ), width b w Height b h The object confidence score and the conditional class probabilities C1, C2, ..., C corresponding to each preset category. N .

[0095] Output Tensor: The original output of the detection head is a high-dimensional tensor containing all prediction information from all grid cells. After non-maximum suppression processing, the final output is a structured list, which is the multi-target candidate set.

[0096] For example, in t k At time S1000, after the aforementioned processing flow, the system generates the following multi-target candidate set M. k : M k ={ timestamp: 150000.980, objects:[ {object_id:1,class:'person',confidence:0.95,bbox:[200,300,150,400],is_tracked:true}, {object_id:2,class:'person',confidence:0.88,bbox:[800,320,160,410],is_tracked:false}, {object_id:3,class:'tree',confidence:0.98,bbox:[500,100,200,600],is_tracked:false}] }, Here, `object_id` is a unique identifier assigned by the system through simple temporal association (such as IOU-based tracking). The four values ​​of `bbox` represent the x-coordinate, y-coordinate, width, and height of the top-left corner of the bounding box, respectively. The `is_tracked` flag indicates whether the object is the target currently being tracked by the main computing path. This multi-target candidate set is stored in the memory of the image acquisition unit 34 and dynamically refreshed at a frequency of 30Hz.

[0097] S1200: Parsing Target Switching Commands Based on Remote Control Operation Vectors In target pre-control mode, not all touch swipes are directly converted into gimbal angular velocity commands. Instead, they are classified according to their motion characteristics, such as the instantaneous speed and duration of the swipe gesture. Swipes with long duration and smooth speed are still interpreted as fine-tuning commands for the current tracking target; while swipes with short duration and instantaneous speed exceeding the preset switching speed threshold V are interpreted as commands for fine-tuning the mapping of the target. switch The rapid swipe is interpreted as a target switching instruction.

[0098] For example, when a user performs a quick swipe on the touchscreen from coordinates (1000, 540) to (600, 540), the entire process takes [time]. At this point, the processor calculates its average speed. Pixels per second. Assuming a preset switching speed threshold V. switch 2000 pixels per second, due to This operation was confirmed as a target switching command. The system then generates a switching intent vector: , Here, `direction` is a normalized two-dimensional vector representing the direction of the switching intention (here assumed to be horizontal to the left), and `magnitude` characterizes the urgency of the switching. This intention vector It is encoded and transmitted to the image acquisition device 34 via a wireless link.

[0099] S1300: Target Selection and Switching Trajectory Planning Image acquisition device 34 receives the intent vector Then, the target optimization algorithm is immediately executed, which selects the target from the current multi-target candidate set M. k In this process, based on the switching intention, the most suitable target that is not currently being tracked is found as the new tracking target. Specifically, a switching score is calculated for each candidate target, and the orientation matching degree is calculated if and only if the position vector magnitude of the candidate target is not zero.

[0100] Among them, the switching score S cand The calculation is as follows: , in: F dir It is a direction matching function used to evaluate candidate targets. Relative to the current goal Position vector and intention vector The direction of consistency, the value of this function is in Between them, the value is 1 when the directions are completely consistent; Instead, it maps the user's physical actions (such as swiping left on the screen) to selection intents within the scene. Here, a vector similarity metric is used to map the user's switching intent vector. The position vector of the current target pointing to the candidate target Compare them. At this point, F... dir The higher the value, the closer the position and orientation of the candidate target match the user's operational intent, and so on; generally, F dir It is usually denoted as the dot product of two unit vectors, i.e. The max(0, ...) here is to directly set the negative values ​​representing opposite directions in the dot product result to 0, which makes the direction matching score of any candidate target behind the user's operation direction directly zero and not considered.

[0101] F dist It is a distance penalty function used to penalize candidate objects that are too far from the current target, reducing their value to a certain value. d is the Euclidean distance between the current target and the candidate target in pixel coordinates, and k is the scaling factor; the value of k depends on the screen size, and is usually set to the reciprocal of the screen's diagonal pixel length. This is to normalize the distance, ensuring consistent performance across devices with different resolutions. Adding a "1" prevents a division-by-zero error when the distance d is 0; while This indicates that, based on directional intent, operators typically tend to switch to objects closer to the current target, which aligns with the natural shift in visual attention. Therefore, F... dist Setting it as the reciprocal function of distance allows for a lower score the farther away the target is, thus penalizing distant candidate targets. This not only improves the intelligence of target selection but also ensures the visual effect and execution efficiency of the final generated trajectory from the bottom up.

[0102] C cand This represents the detection confidence of the candidate object; in real-world complex scenarios, the object detection algorithm may output multiple candidate boxes, but the confidence C of each candidate box is different. cand Not entirely the same. A low-confidence target is likely a false positive or severely occluded; switching to such a target can easily lead to tracking failure. Therefore, [the following is a possible interpretation:] As part of the scoring, it can effectively filter out low-quality candidate targets, ensuring that the system only switches between clear targets, thereby improving the robustness of the system.

[0103] By integrating these three factors, the system can adjust the importance of each dimension according to the needs of different scenarios. For example, in scenarios requiring rapid response, the weight of directional matching can be increased; while in scenarios pursuing aesthetic camera movement, the weight of distance penalty can be appropriately increased, and so on.

[0104] Easy to understand, w dir ,w dist ,w conf These are preset weighting coefficients, and their sum is 1.

[0105] For example, the aforementioned multi-objective candidate set M continues to be used. k and the intent vector for switching to the left Assume the current target being tracked is object_id=1, with its center coordinates at (275, 500), and the candidate targets are object_id=2 (human) and object_id=3 (tree).

[0106] For object_id=2 (human), its center coordinates are (880, 525), and its position vector relative to object_id=1 is [605, 25]. The dot product of this vector with the intended direction [-1, 0] is negative, indicating a direction mismatch. F dir It is close to 0.

[0107] For object_id=3 (tree), since its category is 'tree', it is preset as a non-switchable target type in the target selection logic, and its switching score is directly set to 0. .

[0108] Suppose there is another candidate object_id=4 in the scene, with center coordinates (-50, 510) and a position vector relative to the current target of [-325, 10], which is highly consistent with the intention direction [-1, 0]. F dir A value close to 1 indicates a confidence level of 0.92. The system will calculate its switching score and compare it with all other candidate objects in the scene that meet the criteria. Ultimately, the candidate object with the highest switching score will be selected as the new tracking target, denoted as T. new And so on.

[0109] After identifying the new tracking target T new Then, the system will plan a path from the current gimbal attitude (corresponding to the old target T) current ) to the target pose (corresponding to the new target T) newThe trajectory of the motion. In order to ensure a smooth visual effect, the trajectory is usually planned as a fifth-order polynomial or a Bézier curve, where the velocity and acceleration of the curve are zero at the start and end points, thus achieving a smooth start and stop.

[0110] S1400: Trajectory Correction and Pre-control Command Generation After generating the initial motion trajectory, the system performs trajectory safety verification and correction. Here, obstacle information from the multi-target candidate set is used to assess the collision risk of the planned trajectory.

[0111] Specifically, the system performs discrete-time sampling along the planned motion trajectory, and at each sampling point t... i The image coordinates (x, y) that the center of the gimbal's field of view is expected to point to at that moment are calculated. i ,y i Then, the system checks whether the coordinates fall within the multi-target candidate set M. k Marked as obstacles (e.g.) The bounding box of the object.

[0112] Therefore, the system constructs an obstacle avoidance risk function Cost. obs Input the coordinates (x, y) of the view center into the model to output a non-negative obstacle avoidance risk value.

[0113] It should be noted that, in order to effectively reflect changes in risk, the reciprocal of the squared distance is used here. This way, when the coordinates are far away from all obstacles, the cost is 0; as the coordinates get closer to or into the obstacle bounding box, the obstacle avoidance risk increases exponentially. This exponentially changing nonlinear characteristic provides a clear gradient guide for subsequent trajectory correction, forcing the corrected trajectory to shift towards the open area with the least risk. Meanwhile, in actual calculations, if the view center coordinates (x, y) happen to fall on the obstacle boundary (i.e., distance d = 0), the traditional reciprocal function will cause a division-by-zero anomaly (i.e., the denominator is zero), leading to algorithm crashes or the trajectory deviating infinitely. To prevent this anomaly, a small positive constant can be introduced into the denominator. This ensures that even in the event of a collision, the function output remains a computable maximal finite value, thus improving the system's robustness. Furthermore, for multi-objective scenarios, it's crucial to consider the risks of all potential obstacles in the scene, not just a single obstacle. Therefore, this approach employs... Risk accumulation is implemented to prevent the gimbal trajectory from accidentally entering the collision zone of another obstacle while avoiding a single obstacle. Specifically: , Where d is the shortest distance from the point to the bounding box. It is a very small positive number, which can be set to 10 here.-6 .

[0114] The system will then calculate the total obstacle avoidance risk for the entire trajectory. If the total obstacle avoidance risk exceeds a preset safety threshold (Cost), the system will proceed accordingly. max This means that the switching trajectory has a high risk of visual occlusion. In this case, trajectory correction is required, which involves superimposing a correction displacement on the original trajectory, such as superimposing a smooth waveform motion composed of a sine function on the pitch axis to bypass the obstacle.

[0115] The corrected trajectory will be evaluated for cost again. This process will be repeated until a safe trajectory is found after the number of iterations reaches the preset maximum safe iteration count (e.g., 10). In this case, the system will abandon the switch and issue a visual alarm to the user.

[0116] For example, suppose the straight path from target 1 to target 4 briefly crosses the bounding box of object_id=3 (tree). The total obstacle avoidance risk of the initial trajectory is calculated to be 15.7, which is higher than the safety threshold of 10.0. At this point, the system superimposes a small pitch axis motion on the original yaw axis motion: , Where A is the maximum elevation (e.g., 0.5 degrees), and T total The total switching time is denoted as . After correction, the new trajectory's projection on the image will bypass the tree trunk, reducing its total obstacle avoidance cost to 2.1, which is below the threshold, achieving precise obstacle avoidance, and so on.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0118] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0119] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0120] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0121] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0122] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for wireless networking data transmission using a separate handheld gimbal, characterized in that, The method is applied to a detachable handheld gimbal, which includes a follow-shot drive component, a remote control component, and an image acquisition component. The remote control component is communicatively connected to the follow-shot drive component and the image acquisition component via a wireless communication network. The follow-shot drive component drives the image acquisition component to move to a target position. Upon receiving a modality activation command, the current modality is switched to the target pre-control modality, and a multi-target candidate set is determined based on the processed target environment image; wherein, the target environment image is acquired by the image acquisition device; and the multi-target candidate set is obtained based on a deep learning target detection model. Obtain a switching intent vector, wherein the switching intent vector is obtained by mapping the original touch command of the remote control device. The switching intent vector consists of vector type, switching direction and switching urgency. The switching direction is represented by the sliding direction of the original touch command, and the switching urgency is represented by the sliding speed of the original touch command. Based on the switching intent vector and the multi-target candidate set, a switching score is calculated to determine a new tracking target; wherein, the switching score calculation includes: The process involves calculating the difference between the candidate target's position vector and the current target's position vector, then calculating the dot product between the difference and the switching intent vector. Based on this dot product, a direction matching function is used to determine the consistency value between the difference and the switching intent vector. This consistency value is then multiplied by the corresponding weight coefficient. The consistency value is set to zero if the directions of the difference and the switching intent vector are opposite or if the dot product is zero. A distance penalty function is used to determine the distance penalty between the candidate target's position vector and the current target's position vector. This distance penalty value is then multiplied by the corresponding weight coefficient. Finally, the detection confidence of the candidate target is multiplied by the corresponding weight coefficient. The three product results are then summed to obtain the switching score. Based on the new tracking target, the target motion trajectory is planned, which includes the motion trajectory of the follow-up driving component from the current posture to the corresponding posture of the new tracking target.

2. The method according to claim 1, characterized in that, After switching the current mode to the target pre-controlled mode, the method further includes: Based on the obstacle targets in the multi-target candidate set, an obstacle avoidance risk function is constructed; The motion trajectory is subjected to security verification; If the obstacle avoidance risk value of the motion trajectory exceeds the risk threshold, the motion trajectory is corrected to generate an obstacle avoidance motion trajectory.

3. The method according to claim 1, characterized in that, The method further includes: In touch mode, the original touch command is obtained, and a remote control operation vector is obtained based on the original touch command. The remote control operation vector includes a synchronization timestamp; and the remote control operation vector corresponds to the operation action performed on the remote control device. The remote control operation vector is broadcast to the tracking drive component and the image acquisition component via the wireless communication network; The following drive component generates a first motion command based on the remote control operation vector and executes the first motion corresponding to the first motion command; The image acquisition device performs visual feature calculations on the acquired environmental image based on the remote control operation vector and the preset tracking area to obtain a correction vector; wherein, the correction vector includes the acquisition timestamp of the environmental image; The historical motion state of the first motion executed by the tracking drive component at the acquisition timestamp is fused with the correction vector to obtain a target drive command, and a second motion is executed according to the target drive command; the target drive command is used to correct the first motion.

4. The method according to claim 3, characterized in that, The historical motion state of the first motion executed by the tracking drive component at the acquisition timestamp is fused with the correction vector to obtain the target drive command, including: The angular deviation contained in the correction vector is used as the fusion vector; The first filter is used to fuse the fusion vector with the historical motion state to obtain the target driving command.

5. The method according to claim 3, characterized in that, The method further includes: In the visual gesture mode, the image acquisition device acquires image data and parses the image data to obtain the gesture data and position data of the person in the image; The image acquisition device sends the gesture data and the location data to the follow-up driving component; The follow-up driving component adjusts the first movement based on the type of the gesture data to perform a third movement that maintains the following of the person in the image.

6. The method according to claim 3 or 5, characterized in that, Based on the remote control operation vector, the generation of the first motion command includes: Based on a preset first mapping relationship, the screen pixel velocity contained in the remote control operation vector is converted into a target angular velocity; wherein, the target angular velocity is obtained based on a sliding ratio coefficient, which is determined according to the screen pixel velocity scalar, a specific direction component, and the total number of pixels at the resolution; Based on the target angular velocity, a first motion command is generated.

7. The method according to claim 3 or 5, characterized in that, The step of performing visual feature decomposition on the environmental image to obtain the correction vector includes: Based on the remote control operation vector, the region of interest is updated, and the center coordinates of the visual target are determined; Based on the pixel deviation between the center coordinates and the optical center of the image, and the preset intrinsic parameter matrix of the image acquisition device, the pixel deviation is converted into an angle deviation; The correction vector is generated based on the angular deviation.

8. A separate handheld gimbal wireless networking system, characterized in that, Includes a detachable handheld gimbal, the detachable handheld gimbal comprising: Remote control unit, used to acquire original touch commands in touch mode; The image acquisition device is connected to the remote control device via a wireless communication network and is used to acquire images of the target environment; The tracking drive component communicates with the remote control unit via a wireless communication network. Upon receiving a modal activation command, the system switches the current modality to the target pre-control modality and determines a multi-target candidate set based on the processed target environment image; wherein the multi-target candidate set is obtained based on a deep learning target detection model; a switching intent vector is obtained, wherein the switching intent vector is obtained by mapping the original touch command of the remote control; the switching intent vector consists of vector type, switching direction, and switching urgency, wherein the switching direction is represented by the sliding direction of the original touch command, and the switching urgency is represented by the sliding speed of the original touch command; based on the switching intent vector and the multi-target candidate set, a switching score is calculated to determine a new tracking target, wherein the switching score calculation includes: The process involves calculating the difference between the candidate target's position vector and the current target's position vector, then calculating the dot product between the difference and the switching intent vector. Based on this dot product, a direction matching function is used to determine the consistency value between the difference and the switching intent vector. This consistency value is then multiplied by the corresponding weight coefficient. The consistency value is set to zero if the directions of the difference and the switching intent vector are opposite or if the dot product is zero. A distance penalty function is used to determine the distance penalty between the candidate target's position vector and the current target's position vector. This distance penalty value is then multiplied by the corresponding weight coefficient. Finally, the detection confidence of the candidate target is multiplied by the corresponding weight coefficient. The three product results are then summed to obtain the switching score. Based on the new tracking target, the target motion trajectory is planned, which includes the motion trajectory of the follow-up driving component from the current posture to the corresponding posture of the new tracking target.

9. The system according to claim 8, characterized in that, The remote control device is also used to obtain a remote control operation vector according to the original touch command, and broadcast the remote control operation vector through the wireless communication network; the remote control operation vector includes a synchronization timestamp, and the remote control operation vector corresponds to the operation action performed on the remote control device; The following drive component is also used to generate a first motion command based on the remote control operation vector, and execute a first motion according to the first motion command; The image acquisition device is also used to perform visual feature calculation on the environmental image based on the remote control operation vector and the preset tracking area to obtain a correction vector; wherein, the correction vector includes the acquisition timestamp of the environmental image; The tracking drive component will also fuse the historical motion state of the first motion executed at the acquisition timestamp with the correction vector to obtain a target drive command, and execute a second motion according to the target drive command; the target drive command is used to correct the first motion.

10. The system according to claim 9, characterized in that, Also includes: In the visual gesture mode, the image acquisition device acquires image data and parses the image data to obtain the gesture data and position data of the person in the image; The image acquisition device sends the gesture data and the location data to the follow-up driving component; The follow-up driving component adjusts the first movement based on the type of the gesture data to perform a third movement that maintains the following of the person in the image.

Images