Remote pan-tilt camera intelligent zoom control method and system based on gesture recognition

Abstract

The application provides a gesture recognition-based intelligent zoom control method and system for a remote gimbal camera, which is applied to a mobile terminal in communication connection with the gimbal camera. The method comprises: acquiring a gesture image sequence; performing gesture recognition through a gesture recognition model at a first frequency to obtain a gesture type; determining a zoom direction according to the gesture type; continuously performing zoom stepping and generating a zoom instruction at a second frequency during the recognition of the gesture type, the stepping value gradually increasing from zero to a preset maximum stepping value with the duration of the gesture type, and the second frequency being independent of the first frequency; and sending the zoom instruction to the gimbal camera to perform a zoom operation; when the gesture type is not recognized, the stepping value is linearly attenuated from a current value to a preset value, and then the zoom stepping is stopped. The application decouples the gesture recognition frequency and the zoom execution frequency, and adjusts the stepping value gradually with the duration of the gesture, so that the zoom process is smoothly started and gently ended, and the requirement for the stability of a remote shooting picture is met.

Description

Technical Field

[0001] This application relates to the field of remote control technology for intelligent devices, specifically to a method and system for intelligent zoom control of a remote gimbal camera based on gesture recognition. Background Technology

[0002] Gimbal cameras are widely used in video shooting, live streaming, security monitoring, and conferencing systems, with zoom being one of their most frequently used functions. In scenarios such as solo video shooting and outdoor live streaming, the user is often the subject or their hands are occupied, requiring remote control of the gimbal camera's zoom without physically touching the device. Gesture control provides a natural interaction method for these scenarios. The goal of gesture zoom control is to make the focal length of the image change smoothly and continuously according to the user's intention.

[0003] However, in existing gesture zoom control methods, the gesture recognition result directly triggers and terminates the zoom action, resulting in a step change in zoom magnification at the start and stop times, causing a perceptible abrupt jump and pause in the image. Simultaneously, fluctuations in the gesture recognition frame rate directly translate into fluctuations in the zoom execution rhythm, making the zoom process inconsistently fast and slow. These phenomena lead to uneven image changes and abrupt starts and stops during gesture zoom, failing to meet the image stability requirements of remote shooting scenarios. Summary of the Invention

[0004] To address the shortcomings of existing gesture-based zoom controls, such as abrupt zoom start / stop and uneven image transitions, which fail to meet the image stability requirements of remote shooting scenarios, this application provides a gesture recognition-based intelligent zoom control method and system for remote gimbal cameras. This method and system trigger zoom control through gesture image recognition, decoupling the gesture recognition frequency from the zoom execution frequency. During the duration of the gesture, the method and system gradually increase the zoom step value from zero to a preset maximum step value, and then continuously execute the zoom step at a second frequency; when the gesture disappears, the step value linearly decays to a preset value and then automatically stops. This mechanism enables smooth, natural start / stop remote zoom control. Specifically, this application provides the following technical solutions: In a first aspect, this application provides a remote gimbal camera intelligent zoom control method based on gesture recognition, applied to a mobile terminal, wherein the mobile terminal is communicatively connected to the gimbal camera, and the method includes: Obtain the gesture image sequence; The gesture image sequence is subjected to gesture recognition at a first frequency through a gesture recognition model to obtain the gesture type, which is either a first gesture type or a second gesture type. The zoom direction is determined based on the gesture type, and the zoom direction is either a first zoom direction or a second zoom direction. During the recognition of the gesture type, zoom steps are continuously executed at a second frequency, and zoom commands are generated according to the zoom direction. The second frequency is independent of the first frequency. The step value of the zoom step is dynamically adjusted according to the duration of the gesture type. When the duration is less than a preset acceleration duration, the step value linearly increases from zero to a preset maximum step value. After the duration reaches the preset acceleration duration, the step value remains at the preset maximum step value. The first frequency is lower than the second frequency. The zoom command is sent to the gimbal camera to cause the gimbal camera to perform a zoom operation; When the gesture type is not recognized, the step value decreases linearly from the current value to a preset value, and the zoom stepping stops.

[0005] Optionally, during the process of the step value linearly decaying from the current value to the preset value, when the gesture type is re-identified, the decay is stopped, and the step value linearly increases again from the current value during the decay process to the preset maximum step value.

[0006] Optionally, during the process of the step value linearly decaying from the current value to the preset value, when the re-identified gesture type is different from the gesture type before entering the deceleration phase, the step value linearly increases again from the current value during the decay process to the preset maximum step value, and at the same time, the zoom direction is switched to the zoom direction corresponding to the re-identified gesture type. Optionally, when the gesture type is not recognized, the step value linearly decreases from the current value to the preset value within a preset deceleration time, wherein the preset deceleration time is less than the preset acceleration time.

[0007] Optionally, the first gesture type corresponds to a gesture with all fingers spread, and the second gesture type corresponds to a gesture with some fingers spread.

[0008] Optionally, performing gesture recognition on the gesture image sequence at a first frequency includes: Hand key points are detected based on the gesture images in the gesture image sequence; The opening and closing state of each finger is determined based on the key points of the hand. The gesture type is identified based on the combination of the opening and closing states of each finger; wherein, when the combination of the opening and closing states is the same, the gesture type is distinguished based on a distance threshold between the fingertip of the thumb and the fingertip of the index finger.

[0009] Optionally, determining the opening and closing state of each finger based on the key points of the hand includes: Determine the angle between the direction vectors of adjacent phalanges based on the key points of adjacent phalanges of each finger; The angle between the direction vectors is compared with a preset angle threshold. If the angle between the direction vectors is less than the preset angle threshold, the corresponding finger is determined to be in an open state.

[0010] Optionally, determining the zoom direction based on the gesture type includes: When the gesture type changes, after continuously recognizing the changed gesture type within a preset time window, the zoom direction is switched to the zoom direction corresponding to the changed gesture type; wherein, the first zoom direction corresponds to increasing the zoom magnification, and the second zoom direction corresponds to decreasing the zoom magnification.

[0011] Optionally, when the zoom magnification corresponding to the zoom step reaches the maximum zoom magnification or the minimum zoom magnification, the zoom step is stopped.

[0012] Secondly, this application provides a gesture recognition-based intelligent zoom control system for a remote gimbal camera, comprising: Gesture acquisition module, used to acquire gesture image sequences; A gesture recognition module is used to perform gesture recognition on the gesture image sequence at a first frequency to obtain a gesture type, wherein the gesture type is a first gesture type or a second gesture type. A zoom control module is configured to determine the zoom direction based on the gesture type, continuously execute zoom steps at a second frequency and generate zoom commands based on the zoom direction while the gesture type is recognized, and linearly decay the step value from the current value to a preset value and stop executing the zoom steps when the gesture type is not recognized; wherein, the zoom step value is dynamically adjusted according to the duration of the gesture type, when the duration is less than a preset acceleration duration, the step value linearly increases from zero to a preset maximum step value, and after the duration reaches the preset acceleration duration, the step value remains at the preset maximum step value; the zoom direction is either a first zoom direction or a second zoom direction, and the second frequency is independent of the first frequency; the first frequency is lower than the second frequency. The communication module is used to send the zoom command to the gimbal camera; A gimbal camera is used to receive the zoom command and perform the zoom operation.

[0013] Beneficial effects: 1. This application triggers the zoom control of the gimbal camera through gesture image recognition. Users can remotely control the zoom without touching the operating device. The zoom adjustment can still be completed independently when the user is the subject or when both hands are occupied.

[0014] 2. By decoupling the gesture recognition frequency from the zoom execution frequency, gesture recognition can drive a higher frequency zoom step by running at a lower frequency, and fluctuations in the recognition frequency do not affect the smoothness of the zoom process. 3. By dynamically adjusting the zoom step value according to the duration of the gesture, the zoom step value gradually increases from zero to the preset maximum step value after the gesture is recognized, ensuring a smooth start to the zoom process and avoiding abrupt jumps when the screen starts. When no gesture type is recognized, the zoom step value linearly decreases from the current value to the preset value before zooming stops, ensuring a smooth finish to the zoom process and avoiding abrupt pauses when the screen stops. At the same time, the zoom direction is switched only after confirmation through a time window when the gesture type changes, avoiding frequent direction switching caused by gesture jitter and improving the stability of zoom control.

[0015] In summary, this application makes the image changes during gesture zoom smooth and continuous, with natural start and stop, thus meeting the requirements for image stability in remote shooting scenarios. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall process of a remote gimbal camera intelligent zoom control method based on gesture recognition, provided in an embodiment of this application.

[0017] Figure 2 A schematic diagram showing the angle between key hand points and knuckle vectors provided in an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of the zoom magnification changing over time, provided in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram of the module architecture of a gesture recognition-based remote gimbal camera intelligent zoom control system provided in an embodiment of this application.

[0020] Figure reference numerals: 100, Gesture acquisition module; 200, Image preprocessing module; 300, Gesture detection module; 400, Gesture recognition module; 500, Gesture state management module; 600, Zoom command generation module; 700, Zoom execution module; 800, Communication module; 900, Gimbal camera; 1000, Gesture frequency statistics module. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0023] Gimbal cameras are widely used in video shooting, live streaming, security monitoring, and conferencing systems. After connecting to the gimbal camera via a mobile device, users can remotely framing, shooting, and zooming. In scenarios such as solo video shooting and outdoor live streaming, users need to adjust the gimbal camera's focus without touching the mobile device; gesture control provides a natural interaction method for these scenarios.

[0024] The zoom control method provided in this application can be applied to mobile terminals with image acquisition capabilities, such as smartphones, tablets, and portable shooting devices. The mobile terminal communicates with the gimbal camera wirelessly. Application scenarios include single-person video shooting, outdoor live streaming, conference teaching, and security monitoring. In these scenarios, even when the user is the subject being filmed or their hands are occupied, they can still control the zoom through gestures.

[0025] The control method provided in this application embodiment is executed by a mobile terminal and a gimbal camera. The mobile terminal includes a front-facing camera, a processor, a memory, and a wireless communication component. The front-facing camera captures gesture images of the user, the processor performs gesture recognition and zoom command generation, and the wireless communication component sends the zoom command to the gimbal camera. The gimbal camera includes a communication interface, a lens drive motor, and a zoom mechanism. The communication interface receives the zoom command, and the lens drive motor drives the zoom mechanism to adjust the focal length.

[0026] like Figure 1 As shown, the gesture recognition-based intelligent zoom control method for remote gimbal cameras provided in this embodiment includes the following steps: S100, acquire the gesture image sequence.

[0027] The mobile terminal captures user gesture images via its front-facing camera and outputs a sequence of gesture images at a set capture frequency. The front-facing camera is pointed towards the user; when the user makes a gesture in front of the camera, the camera captures images including the hand area in the form of continuous video frames. The capture process is driven by camera data callbacks. Each captured video frame is output by the mobile terminal to subsequent processing stages, and the video frames are arranged in the order of capture time to form a gesture image sequence.

[0028] For example, the front-facing camera outputs video frames at a capture frequency of 25 frames per second. Each frame is a color image that is 640 pixels wide and 480 pixels high, and the capture interval between two adjacent frames is approximately 40 milliseconds. Each frame in the gesture image sequence contains the user's complete hand area, and the position of the hand area in the image changes with the user's gesture movements and arm position.

[0029] It should be understood that the sampling frequency is related to the recognition response speed and processor load. The sampling frequency can be any value within the range of 4 to 30 frames per second: a lower sampling frequency reduces processor load and power consumption, while a higher sampling frequency improves the response speed to gesture type changes. In this embodiment, the sampling frequency is set to 25 frames per second, achieving a balance between recognition response speed and power consumption. For example, in scenarios sensitive to processor load, such as outdoor strong light, the sampling frequency can be adjusted to 15 frames per second to reduce power consumption; in scenarios requiring higher response speed, the sampling frequency can be adjusted to 30 frames per second.

[0030] The output sequence of gesture images serves as the input for the gesture recognition step, which sequentially performs hand detection and gesture type recognition on each video frame in the sequence.

[0031] Generally, the higher the acquisition frequency, the more gesture images can be recognized per unit time, and the more timely the tracking of user gesture changes. When the acquisition frequency is too low, rapidly changing gestures may be missed between adjacent frames. The acquisition and recognition stages transfer video frames through the processor's internal data buffer. Each frame is consumed by the recognition stage in the acquisition order, without losing frame data.

[0032] The data buffer capacity is related to the acquisition frequency and the processing time. In this embodiment, the data buffer capacity is set to 3 frames, meaning the buffer can temporarily store a maximum of 3 unprocessed video frames. When the processing time of the recognition stage exceeds the acquisition interval, newly acquired video frames enter the buffer to await processing; when the buffer is full, the oldest unprocessed frame is discarded, and the most recently acquired frame overwrites the oldest frame in the buffer, ensuring that the recognition stage always processes the most recent gesture image and avoids the recognition result lagging behind the user's actual gesture due to processing delays.

[0033] For example, when the acquisition frequency is 25 frames / second and the single-frame recognition time is 50 milliseconds, the recognition process requires approximately 1.25 acquisition cycles to process each frame, and the number of frames in the buffer fluctuates between 0 and 2, without triggering frame dropping. When the mobile terminal processor's single-frame recognition time increases to 100 milliseconds due to increased background task load, the recognition process requires approximately 2.5 acquisition cycles to process each frame, and the buffer fills up quickly. After that, each acquired frame discards an older frame, and the recognition process still processes the latest frame at an effective frequency of approximately 10 frames / second, ensuring the timeliness of gesture recognition, and so on.

[0034] In another embodiment, the gesture image sequence can also be captured by a camera separate from the mobile terminal and transmitted to the mobile terminal, where the mobile terminal performs gesture recognition on the received image sequence. For example, in a scenario where the mobile terminal is fixed on a tripod, user gestures can be captured by an external camera and transmitted back to the mobile terminal, expanding the deployment flexibility of gesture capture.

[0035] In this embodiment, the external camera transmits the captured video frames to the mobile terminal via a wired connection (USB) or a wireless connection (WiFi). Upon receiving the video frames from the external camera, the mobile terminal performs the same post-processing steps as the front-facing camera, including image preprocessing, hand detection, key point localization, finger opening / closing determination, and gesture type recognition. The acquisition frequency of the external camera is determined by its own hardware specifications, and the mobile terminal uses the actual output frame rate of the external camera as the primary frequency for gesture recognition.

[0036] For example, in a security monitoring scenario, a PTZ camera is deployed at a high position, a mobile terminal is deployed in the monitoring room, and an external camera is installed near the PTZ camera facing the operator. The operator controls the PTZ camera's zoom by making gestures in front of the external camera. The gesture images captured by the external camera are transmitted via a local area network to the mobile terminal in the monitoring room for gesture recognition and zoom command generation. In a conference scenario, an external camera can be installed in front of the conference table, facing the speaker. The speaker controls the zoom of the PTZ camera in the conference room by making gestures on the stage, without having to leave the stage to operate the mobile terminal, and so on.

[0037] S200, perform gesture recognition on the gesture image sequence at a first frequency through a gesture recognition model to obtain the gesture type, wherein the gesture type is a first gesture type or a second gesture type.

[0038] In this embodiment, the aforementioned first frequency is specifically the gesture recognition frequency, that is, the frequency at which gesture recognition is performed on the gesture image sequence. Gesture recognition is driven by camera data callbacks, and its calling frequency is consistent with the frame rate of the mobile terminal's front-facing camera, which in this embodiment is approximately 4 to 30 frames per second.

[0039] Specifically, the gesture recognition process includes the following sub-steps: S210, detect key points of the hand based on the gesture images in the gesture image sequence.

[0040] The mobile terminal uses a gesture recognition model to perform hand detection on each frame of a gesture image sequence. The gesture recognition model includes a palm detection model and a key point localization model. The palm detection model is used to locate the hand region in the image, and the key point localization model is used to locate key points of the hand within the hand region.

[0041] In this embodiment, the hand detection model uses the BlazePalm model, and the key point localization model uses the Landmark model. Both are lightweight models based on deep learning and can run locally on the mobile terminal.

[0042] In another implementation, the gesture recognition model can use the MediaPipeHands framework instead of the combination of BlazePalm and Landmark.

[0043] The MediaPipeHands framework also outputs the normalized coordinates of 21 key points on the hand. The numbering order and meaning of the key points are consistent with those in this embodiment, and the subsequent finger opening and closing determination and gesture type recognition steps do not need to be modified.

[0044] In another embodiment, the gesture recognition model can adopt the OpenPose hand key point detection model. The number and number of key points output by the OpenPose hand model are the same as in this embodiment, which is suitable for scenarios with sufficient processor computing power.

[0045] It should be noted that the selection of the above alternative models is based on the model's inference speed and the computing power of the mobile terminal: the combination of BlazePalm and Landmark has a single-frame inference time of approximately 10 to 30 milliseconds, suitable for real-time processing scenarios on mobile terminals; MediaPipeHands has a single-frame inference time comparable to BlazePalm and provides more comprehensive tracking optimization, suitable for continuous gesture recognition scenarios; OpenPose has a longer single-frame inference time, suitable for terminals with sufficient computing power or non-real-time scenarios. Regardless of the gesture recognition model used, its output is the normalized coordinates of 21 key points of the hand, and the subsequent finger opening / closing state determination and gesture type recognition logic are not affected by the model selection.

[0046] The hand detection model performs detection on the gesture image and outputs a hand detection bounding box. The detection result includes the center coordinates, width, height, and detection confidence score of the bounding box, and also outputs the initial coordinates of 7 hand key points. The mobile terminal filters the detection confidence score. When the detection confidence score is lower than a preset confidence threshold, it is determined that no valid hand has been detected in the current frame, the detection result of that frame is discarded, and subsequent key point localization and gesture recognition are not performed.

[0047] The preset confidence threshold can be adjusted within the range of 0.5 to 0.9; in this embodiment, it is set to 0.7. This value is based on the following: if the confidence threshold is too low, non-hand objects in the background may be falsely detected as hands, leading to incorrect results in subsequent gesture recognition; if the confidence threshold is too high, some valid hand detection results may be falsely filtered out in scenarios where the hand is far from the camera or in insufficient lighting. The value of 0.7 strikes a balance between the false detection rate and the false negative rate. When no valid hand is detected for several consecutive frames, the mobile terminal determines that the user has not made a gesture, and the zoom direction state is set to stop.

[0048] For example, for a gesture image that is 640 pixels wide and 480 pixels high, the center of the palm detection box has an x-coordinate of 320 pixels, a y-coordinate of 240 pixels, a width of 180 pixels, a height of 200 pixels, and a detection confidence score of 0.95.

[0049] After detecting the palm area, the mobile terminal normalizes the palm area into a standardized image of 224 pixels wide and 224 pixels high through affine transformation. This image is then input into the keypoint localization model, and the normalized coordinates of 21 keypoints on the hand are output. These 21 keypoints are arranged in a fixed order: keypoint 0 is the base of the wrist; keypoints 1 to 4 are the joints of the thumb; keypoints 5 to 8 are the joints of the index finger; keypoints 9 to 12 are the joints of the middle finger; keypoints 13 to 16 are the joints of the ring finger; and keypoints 17 to 20 are the joints of the little finger. The coordinates of each keypoint are normalized to the range of 0 to 1, and so on.

[0050] S220, determine the opening and closing state of each finger based on the key points of the hand.

[0051] The mobile terminal uses a finger vector direction consistency algorithm based on the location of 21 key points to determine whether each finger is in an open state. For example... Figure 2 As shown, for each finger, the direction vector between adjacent phalanges is calculated based on the key points corresponding to each phalanx of that finger, and the angle between adjacent direction vectors is calculated. The opening and closing state of the finger is determined based on the comparison result between the angle and the preset angle threshold.

[0052] For example, for the index finger, its key point sequence is the base of the palm, proximal phalanx, middle phalanx, distal phalanx, and fingertip, denoted as P0, P1, P2, P3, and P4 respectively. The mobile terminal calculates the direction vectors for each segment: V0 is the difference vector between P1 and P0, V1 is the difference vector between P2 and P1, V2 is the difference vector between P3 and P2, and V3 is the difference vector between P4 and P3, and calculates the angle between adjacent direction vectors V0 and V1. The angle between V1 and V2 The angle between V2 and V3 .

[0053] The angle between adjacent direction vectors is calculated using the following formula:

[0054] In the formula, Direction vector and dot product, and Direction vectors and The length of the mold, for and The angle between adjacent phalanges is expressed in radians. When the fingers are extended, the direction vectors of adjacent phalanges tend to be consistent, and the angle approaches 0; when the fingers are bent, the difference in direction between the direction vectors of adjacent phalanges increases, and the angle increases.

[0055] Among them, the preset angle threshold Used for determining finger opening and closing: when the angle between adjacent direction vectors is less than... When the angle is greater than or equal to the specified angle, the finger is determined to be in an open state; when the angle is greater than or equal to the specified angle, the finger is determined to be in an open state. When the finger is closed, it is determined that the finger is in a closed state.

[0056] In this embodiment, The value of affects recognition sensitivity and the risk of false touches: the larger the value, the more the finger is judged as open even when it is bent to a greater degree, the higher the recognition sensitivity, and the higher the probability of misjudging a non-target gesture as a target gesture; the smaller the value, the more the finger needs to be stretched straighter to be judged as open, the lower the recognition sensitivity, and the lower the false touch rate. It can be adjusted within the range of 15 to 40 degrees; here we take 25 degrees to strike a balance between recognition sensitivity and false touch rate.

[0057] Because the anatomical structure of the thumb differs from that of the other four fingers, its movement pattern is mainly abduction and adduction, while the other four fingers are mainly flexion and extension. Therefore, in calculating the direction vector of the thumb, the starting point is not the key point at the base of the palm, but the key point at the connection between the base of the thumb and the palm, i.e., key point 1.

[0058] For example, the keypoint sequence of the thumb is keypoints 1, 2, 3, and 4, corresponding to the base of the thumb, the proximal phalanx, the distal phalanx, and the fingertip, respectively. Since the thumb only has three phalanges, only the two direction vectors V1 (the difference vector between keypoints 2 and 1) and V2 (the difference vector between keypoints 3 and 2) are calculated, along with the angle between them. The preset angle threshold for the thumb can be set independently; here, 30 degrees is chosen to accommodate the thumb's abduction and adduction movements.

[0059] The following section provides a complete explanation of the calculation process for the angle between the direction vectors in determining finger opening and closing, using specific numerical examples.

[0060] For example, the normalized coordinates of the key points of the index finger are: P0=(0.45,0.85), P1=(0.48,0.70), P2=(0.50,0.55), P3=(0.51,0.42), P4=(0.52,0.30).

[0061] Calculate the direction vector for each segment: =P1-P0=(0.03,-0.15), =P2-P1=(0.02,-0.15), =P3-P2=(0.01,-0.13), =P4-P3=(0.01,-0.12).

[0062] calculate and The included angle: =0.03 0.02 + (-0.15) (-0.15) = 0.0006 + 0.0225 = 0.0231 = = =0.1530, = = =0.1513, =0.0231 / (0.1530 0.1513) = 0.0231 / 0.02315 = 0.9978 = =3.8 degrees. Due to =3.8 degrees less =25 degrees, the segment is deemed acceptable.

[0063] Calculate similarly and The included angle and and The included angle If the angle between all adjacent direction vectors is less than 1 / 2 0 ... If the index finger is open, then it is determined that the index finger is in an open state.

[0064] In this example, the direction vectors of each segment of the index finger tend to be consistent, and the angle between each segment is less than 25 degrees, indicating that the index finger is in a straight state. If the user bends the index finger, the ordinates of P3 and P4 will increase (bending towards the palm). and The direction will deviate and The direction, the angle between adjacent vectors increases to more than The index finger is used to determine that the finger is in a closed state.

[0065] In addition, during the finger opening / closing determination process, the mobile terminal checks the validity of key point coordinates. When the normalized coordinates of a hand key point exceed the valid range of 0 to 1, it indicates that the key point is located outside the image boundary, possibly due to partial occlusion or removal of the hand from the frame. For key points with coordinates outside the valid range, the mobile terminal crops them to the boundary values ​​of the valid range and marks the key point as low confidence. When a finger's key points contain a low confidence marker, the finger's opening / closing state determination result is marked as uncertain and it is not included in the gesture type recognition.

[0066] For example, when the user's little finger moves out of the frame, the key point coordinates of the distal phalanx and fingertip of the little finger may be outside the normalization range. In this case, the open / closed state of the little finger is determined to be uncertain, and the gesture type is identified only based on the open / closed states of the thumb, index finger, middle finger, and ring finger. When the number of uncertain fingers exceeds two, the mobile terminal determines that the gesture recognition result of the current frame is unreliable, and the entire frame's recognition result is discarded, which is equivalent to no gesture type being recognized.

[0067] S230, the gesture type is identified based on the combination of the opening and closing states of each finger.

[0068] The mobile terminal recognizes the gesture type based on the combination of the open and closed states of the five fingers. In this embodiment, the first gesture type is specifically a gesture with all fingers spread out, i.e., a palm gesture with the palm facing the camera; the second gesture type is specifically a gesture with some fingers spread out, such as a scissor gesture or an OK gesture.

[0069] For example, when the thumb, index finger, middle finger, ring finger, and little finger are all open, it is recognized as a slap gesture; when only the index finger and middle finger are open and the other fingers are closed, it is recognized as a scissor gesture; when the thumb and little finger are open and the other fingers are closed, it is recognized as a 666 gesture; when only the thumb is open and pointing upwards, it is recognized as a thumbs-up gesture.

[0070] When multiple hands are present in the image, the mobile terminal counts the gesture types of all detected hand objects and takes the gesture type with the highest frequency as the current recognition result.

[0071] For example, if two hands are detected in the image, one is identified as a palm gesture and the other as a scissor gesture, then the current recognition result is the one that appears more frequently; if both hands are palm gestures, then it is further determined to be a double palm gesture.

[0072] For the double-palm gesture, the mobile terminal takes the average position of all detected objects as the gesture position. The gesture type output by the mobile terminal is represented by a number, for example, the slap gesture is type 0, the scissor gesture is type 1, the OK gesture is type 2, and so on.

[0073] It should be noted that in multi-hand scenarios, the specific process by which the mobile terminal performs aggregate statistics on the gesture types of each hand object is as follows: The mobile terminal maintains a gesture type count array, the length of which is equal to the total number of predefined gesture types (7 in this embodiment). Each element of the array records the number of times the corresponding gesture type is detected in the current frame. When the palm detection model outputs multiple detection boxes, the mobile terminal sequentially performs keypoint localization, finger opening / closing determination, and gesture type recognition for each detection box, and accumulates the recognition results into the count array. After all detection boxes have been processed, the mobile terminal selects the gesture type with the largest count value from the count array as the final recognition result for the current frame. When two gesture types have the same count value, the mobile terminal prioritizes selecting the gesture type that is the same as the recognition result of the previous frame to maintain the temporal continuity of gesture recognition.

[0074] For example, when three hand objects are detected in the image, two of which are identified as slap gestures and one as a scissor gesture, the count value of the slap gesture is 2, the count value of the scissor gesture is 1, and the final recognition result is the slap gesture.

[0075] S240, when the combination of the opening and closing states is the same, the gesture type is distinguished according to the distance threshold between the fingertip of the thumb and the fingertip of the index finger.

[0076] The OK gesture and the figure-eight gesture have the same combination of open and closed fingers: the thumb and index finger are open, and the other fingers are closed. They cannot be distinguished by their open and closed state alone. Mobile terminals differentiate the two gestures by calculating the spatial distance between the tips of the thumb and index finger.

[0077] Let the key point at the tip of the thumb be key point 4, and the key point at the tip of the index finger be key point 8. The Euclidean distance between the two points is calculated using the following formula:

[0078] In the formula, The coordinates of the thumb tip. The coordinates are for the fingertip of the index finger, and the coordinate values ​​have been normalized to the range of 0 to 1.

[0079] When distance Less than the preset distance threshold When the distance is [not specified], it is considered an OK gesture, meaning the tips of the thumb and index finger touch to form a circle; when [not specified], it is considered an OK gesture. Greater than or equal to When the hand gesture is determined to be the figure eight, it means that the thumb and index finger are separated to form a figure eight shape.

[0080] Generally, The value can be adjusted within the range of 0.05 to 0.12; here, 0.08 is chosen. The basis for this value is as follows: when a user makes an OK gesture, the tips of the thumb and index finger are physically touching or very close to each other, and the normalized distance is usually less than 0.05; when a user makes a figure-eight gesture, the distance between the two fingertips in the normalized coordinate system is usually greater than 0.10. 0.08 is in between and can effectively distinguish between the two gestures. A larger value makes it easier to identify as the OK gesture, while a smaller value makes it easier to identify as the figure-eight gesture.

[0081] In practical use, the degree of contact between the tip of the thumb and the tip of the index finger when a user makes an OK gesture varies from person to person. Users with thicker fingers may experience a greater normalized distance at the fingertip contact point than users with thinner fingers. To improve the stability of the distinction, the mobile terminal performs a moving average filter on the distance values ​​from multiple consecutive frames: The mobile terminal maintains a sliding window of 5 frames in length, and calculates the distance in each frame. Store the distance in a window, and take the arithmetic mean of all distance values ​​within the window as the filtered distance. ,use and The gesture type is determined by comparison. The moving average filter aims to eliminate the instantaneous impact of single-frame key point positioning errors on the distance value.

[0082] For example, the distance values ​​for five consecutive frames are 0.06, 0.04, 0.09, 0.05, and 0.06, respectively. The average distance after filtering is 0.06, which is less than... =0.08, which is determined to be the OK gesture. Without filtering, the distance value of 0.09 in the third frame is greater than 0.08, and this frame will be misjudged as the figure-eight gesture, causing the gesture recognition result to frequently switch between the OK gesture and the figure-eight gesture.

[0083] The gesture recognition process is repeated at a first frequency, and the gesture type obtained each time is used as the basis for determining the zoom direction. When gesture recognition does not detect any hand, the recognition result is empty, that is, no gesture type was recognized.

[0084] S300, determine the zoom direction based on the gesture type, wherein the zoom direction is a first zoom direction or a second zoom direction.

[0085] The gesture type output by the gesture recognition step is used to determine the zoom direction of the gimbal camera. In this embodiment, the first zoom direction is specifically the magnification direction, i.e., the direction of increasing the zoom magnification; the second zoom direction is specifically the zoom reduction direction, i.e., the direction of decreasing the zoom magnification. The mapping relationship between gesture type and zoom direction is as follows: the first gesture type is mapped to the first zoom direction, and the second gesture type is mapped to the second zoom direction.

[0086] For example, when a slap gesture is detected, the zoom direction is determined to be the magnification direction; when a scissor gesture or OK gesture is detected, the zoom direction is determined to be the reduction direction.

[0087] The mapping relationship between the aforementioned gesture types and zoom directions can be customized by the user. The mobile terminal provides a mapping configuration interface, allowing users to map any gesture type to either the first or second zoom direction. For example, for users who habitually use the peace sign to indicate zooming, the peace sign can be mapped to the first zoom direction, and the slap gesture to the second zoom direction. The system's zoom execution logic is unaffected by changes to the mapping relationship; the zoom control module only executes zoom steps based on the zoom direction state, which is determined by both the mapping relationship and the current gesture type, and is not limited here. Changes to the mapping relationship take effect immediately after the configuration is saved, without requiring a restart of the gesture recognition process.

[0088] In one implementation, the mapping relationship is stored in a local configuration file of the mobile terminal. When the gesture recognition process is started, the mobile terminal reads the mapping relationship in the configuration file and determines the zoom direction corresponding to the gesture type based on the mapping relationship.

[0089] It should be noted that gesture recognition and zoom direction determination are performed in each gesture recognition call. The mobile terminal maintains the current zoom direction state, which can be one of three states: first zoom direction, second zoom direction, or stopped. The zoom direction state is updated according to the following rules: if the zoom direction corresponding to the currently recognized gesture type is inconsistent with the current state, it is updated to the zoom direction corresponding to the recognized gesture type; if they are consistent, the current state remains unchanged. Specifically: S310 determines the target zoom direction based on the recognized gesture type.

[0090] The mobile terminal maps the recognized gesture type to a target zoom direction. The target zoom direction is either a first zoom direction or a second zoom direction, where the first zoom direction corresponds to increasing the zoom magnification and the second zoom direction corresponds to decreasing the zoom magnification. The mapping relationship uses the gesture type as the key and the zoom direction as the value.

[0091] For example, when the gesture type is a slap gesture, the target zoom direction is the first zoom direction; when the gesture type is a scissor gesture or an OK gesture, the target zoom direction is the second zoom direction. If no gesture type is recognized, no target zoom direction is generated, and the zoom direction state is set to stop.

[0092] S320, when the gesture type changes, the zoom direction is switched to the zoom direction corresponding to the gesture type.

[0093] When the recognized gesture type is different from the previously recognized gesture type, the mobile terminal determines that the gesture type has changed and performs a zoom direction switch. During the switch, if the current zoom direction is different from the target zoom direction, the zoom step along the current direction is stopped, and the zoom direction is updated to the target zoom direction; if the current zoom direction is the same as the target zoom direction, the zoom step value remains unchanged, and the current zoom direction remains unchanged.

[0094] It should be noted that switching the zoom direction only changes the direction of increase or decrease of the current zoom value, and does not change the method of determining the step value. When the gesture type changes, the zoom step value is determined in the following two ways: If the gesture type change occurs during the recognition of the gesture type and the zoom step value is not in the decay process, the duration of the gesture type is accumulated again from zero, and the step value increases linearly from zero to the preset maximum step value as the duration of the re-accumulated duration increases; if the gesture type change occurs during the process of the step value linearly decaying from the current value to the preset value, the decay is stopped, and the step value increases linearly from the current value during the decay process to the preset maximum step value. That is, when switching the zoom direction, the step value remains at the current value at the time of switching and does not return to zero. Only the zoom direction is switched to the zoom direction corresponding to the newly recognized gesture type.

[0095] Taking the gesture change from a palm gesture to a scissor gesture as an example, the current zoom direction changes from the first zoom direction to the second zoom direction, and the zoom execution changes from increasing the zoom magnification to decreasing the zoom magnification. The switching process does not require the user to stop and then start.

[0096] In addition, to avoid frequent direction switching caused by hand tremors, the mobile terminal sets a preset time window for confirmation when switching zoom directions. When the gesture type changes, the mobile terminal records the moment of change. Only after continuously recognizing the changed gesture type within the preset time window will the zoom direction be switched. If the gesture type changes again within the preset time window, the moment of change is recorded again. The gesture type change occurs during the linear decay of the step value from the current value to the preset value. After continuously recognizing the changed gesture type within the preset time window, the decay stops and the zoom direction is switched to the zoom direction corresponding to the changed gesture type.

[0097] It should be noted that the preset time window value is related to the duration of gesture jitter, and in this embodiment, it is set to 200 milliseconds. The technical basis for this value is that when a user makes a gesture switching action, during the transition from one gesture to another, the finger experiences an intermediate transition posture, and gesture recognition may output unstable recognition results during the transition. Actual tests show that the transition posture of the user during gesture switching usually lasts for 100 to 150 milliseconds, and the preset time window of 200 milliseconds covers this transition range and leaves a margin. Generally, if the preset time window value is less than 100 milliseconds, the unstable recognition results in the transition posture may penetrate the image stabilization mechanism, causing the zoom direction to briefly switch and then recover, resulting in perceptible zoom direction jitter; if the value is greater than 500 milliseconds, the user needs to wait a long time after intentionally switching gestures to see the response of zoom direction switching, resulting in a sluggish operation experience. The value of 200 milliseconds strikes a balance between jitter suppression and response speed.

[0098] For example, if a brief user shake causes a frame to be recognized as a peace sign gesture, and the shake lasts for less than 200 milliseconds, the recognition result will not trigger a zoom direction switch; if the user continuously makes the peace sign gesture for more than 200 milliseconds, a switch from the first zoom direction to the second zoom direction will be executed. The preset time window can be adjusted within the range of 100 to 500 milliseconds. The larger the value, the stronger the suppression of shake, but the slower the response of the zoom direction switch.

[0099] S400, continuously execute zoom steps at a second frequency, and generate zoom commands according to the zoom direction, wherein the second frequency is independent of the first frequency.

[0100] In this embodiment, the aforementioned second frequency is specifically the zoom execution frequency, that is, the frequency at which zoom steps are executed. The zoom execution is driven by a timer, which triggers at a fixed second frequency, executing one zoom step each time it is triggered.

[0101] In this embodiment, the independence of the second frequency from the first frequency means that the first frequency is the gesture recognition frequency, which varies with the frame rate of the mobile terminal's front-facing camera, ranging from 4 to 30 frames per second in this embodiment; the second frequency is the zoom execution frequency, which is fixedly set by a timer and is not affected by fluctuations in the first frequency. The gesture recognition stage is responsible for determining the zoom direction and recording whether the gesture type persists, while the zoom execution stage is responsible for continuously executing zoom steps at the second frequency. The two stages operate decoupled.

[0102] It's important to note that the above decoupling setup is necessary because if the gesture recognition frequency were coupled with the zoom execution frequency—meaning a zoom step is executed only after each frame of gesture image is recognized—the zoom execution frequency would fluctuate with the recognition frequency. Furthermore, the output frame rate of a mobile terminal's front-facing camera is not constant under different lighting conditions and processor load (approximately 4 to 30 frames per second). In coupled mode, when the recognition frequency drops to 4 frames per second, the zoom step is executed only once every 250 milliseconds, resulting in perceptible discrete jumps in the zoom process and an unsmooth user experience. When the recognition frequency increases to 30 frames per second, the zoom step is executed once every 33 milliseconds, making the zoom speed approximately 7.5 times faster than at lower frame rates, and the zoom speed becomes uncontrollably variable with the frame rate. By decoupling the settings, the gesture recognition is responsible for setting the zoom direction and monitoring whether the gesture is continuous. The timer independently drives the zoom step at a fixed second frequency. No matter how the gesture recognition frequency fluctuates, the execution frequency of the zoom step is always maintained at 30 times / second. The zoom speed in the constant speed phase is constant at 1.05 times / second, achieving smooth zoom that is independent of the recognition frequency.

[0103] Gesture recognition calls and timer triggers occur independently on the timeline. Gesture recognition calls occur during camera data callbacks, with a call frequency of the first frequency; timer triggers occur independently at a second frequency, executing one zoom step with each trigger. Changes in the gesture recognition frequency do not affect the timer triggering rhythm, ensuring a stable zoom step execution frequency. Specifically: The S410 continuously performs zoom steps at a second frequency.

[0104] The mobile terminal continuously triggers zoom steps at a second frequency via a timer. In this embodiment, the timer is implemented using a system timer, with a trigger frequency of 30 times / second, i.e., triggering once every 33 milliseconds, and executing one zoom step each time.

[0105] In this embodiment, the zoom step value is dynamically adjusted according to the duration of the current gesture type: when a gesture type is recognized, the mobile terminal counts the continuous duration of the gesture type. When the duration is less than the preset acceleration duration, the step value increases linearly from zero to the preset maximum step value as the duration increases. After the duration reaches the preset acceleration duration, the step value remains at the preset maximum step value.

[0106] In addition, the preset acceleration duration is 0.5 seconds, and the preset maximum step value is 0.035 times. If the gesture type change occurs during the recognition of the gesture type and the zoom step value is not in the decay process, the duration of the gesture type will start accumulating from zero again. If the gesture type change occurs during the process of the step value linearly decaying from the current value to the preset value, the decay will be stopped, the duration of the acceleration phase will start counting from zero again, and the step value will continue to linearly increase from the current value when the decay stops to the preset maximum step value, without returning to zero due to the zoom direction switch.

[0107] The direction of the zoom step is determined by the current zoom direction. When the zoom direction is the first zoom direction, the current zoom value increases by the step value; when the zoom direction is the second zoom direction, the current zoom value decreases by the step value.

[0108] The zoom speed is determined by both the step value and the second frequency. During the constant speed phase, the step value remains at the preset maximum step value, and the zoom speed... for:

[0109] in, This is the preset maximum step size, expressed in multiples per step. This is the second frequency, measured in times per second. This represents the zoom speed during the constant speed phase, expressed in times per second.

[0110] For example, when the preset maximum step value is 0.035 times and the second frequency is 30 times / second, the zoom speed during the constant speed phase is: The zoom speed is adjusted by magnification (x / s). The preset maximum step value can be adjusted from 0.02 to 0.08x. A larger preset maximum step value results in faster constant-speed zoom, while a smaller preset maximum step value results in more precise zoom. The preset acceleration time can be adjusted from 0.2 to 1.0 seconds. A shorter preset acceleration time results in faster zooming into the constant-speed phase, while a longer preset acceleration time results in a smoother start-up process. The second frequency can be adjusted from 15 to 60 times / second. A higher frequency results in more frequent zoom steps and a smoother zoom process, but also increases the processor load. No limit is set here.

[0111] S420, Generate zoom command according to the zoom direction.

[0112] The mobile terminal generates corresponding zoom commands based on the current zoom direction. Specifically, when the zoom direction is the first zoom direction, a magnification command is generated to increase the zoom magnification; when the zoom direction is the second zoom direction, a zoom reduction command is generated to decrease the zoom magnification; and when the zoom direction is stopped, a stop command is generated.

[0113] The zoom command is converted by the communication module into a command format that conforms to the gimbal camera's communication protocol. The command format includes fields such as frame header, command type, speed value, and checksum, and the frame tail indicates the end of the command.

[0114] For example, the instruction type field of the zoom-in instruction is 1, the instruction type field of the zoom-out instruction is 2, and the instruction type field of the stop instruction is 0.

[0115] The zoom command data frame includes a frame header, command type, speed value, sequence number, checksum, and frame tail fields. The frame header is used by the gimbal camera receiver to identify the start position of the data frame. The command type field corresponds to three operations: stop, zoom in, and zoom out. The speed value field indicates the zoom step speed level. The sequence number field is an incrementing integer used by the receiver to detect command loss. The checksum field verifies the integrity of the data frame. The frame tail indicates the end of the data frame. After receiving the data, the gimbal camera receiver locates the start position of the data frame, extracts the command type and speed value fields, verifies the checksum, and executes the corresponding zoom operation if the verification passes; otherwise, the frame data is discarded.

[0116] S430 stops zoom stepping when the zoom magnification reaches the maximum or minimum zoom magnification.

[0117] After each zoom step, the mobile terminal checks whether the current zoom value has reached the zoom limit. When the zoom direction is the first zoom direction, if the current zoom value is greater than or equal to the maximum zoom magnification, the zoom step stops and the current zoom value is set to the maximum zoom magnification. When the zoom direction is the second zoom direction, if the current zoom value is less than or equal to the minimum zoom magnification, the zoom step stops and the current zoom value is set to the minimum zoom magnification. In this embodiment, the maximum zoom magnification is 4.0x and the minimum zoom magnification is 1.0x. The maximum zoom magnification can be adjusted according to the gimbal camera lens specifications.

[0118] like Figure 3 As shown, the zoom magnification changes in a step-like manner over time, with the step size gradually increasing during the acceleration phase and remaining constant during the constant-speed phase. When the zoom direction is the first zoom direction, the zoom magnification smoothly starts after an acceleration phase of approximately 0.5 seconds, then increases at a constant speed of 0.035 times every 33 milliseconds until it reaches the maximum zoom magnification of 4.0x and remains thereafter. When the zoom direction is the second zoom direction, the zoom magnification decreases at a constant speed of the same duration after the acceleration phase, until it reaches the minimum zoom magnification of 1.0x and remains thereafter. After reaching the zoom limit, the mobile terminal stops generating zoom commands and sends a stop signal to the zoom execution stage to avoid damage to the lens motor from continuous operation.

[0119] It should be noted that during zooming, the mobile terminal provides a zoom status query interface, and the query results include whether zooming is in progress, the current zoom direction, and the current zoom value.

[0120] Because the gesture recognition and zoom execution processes run in different threads—gesture recognition in the camera data callback thread and zoom stepping in the timer thread—there is concurrent access to the zoom direction state for both processes. The mobile terminal uses a thread synchronization mechanism on the zoom direction state variable to ensure that data contention does not occur between the gesture recognition process writing to the zoom direction and the zoom execution process reading it.

[0121] For example, in the iOS system, the read and write operations of the zoom direction state variable are synchronized through a serial queue: the gesture recognition process submits the zoom direction update operation to the serial queue for execution, and the zoom execution process reads the current zoom direction from the serial queue each time the timer is triggered. The serial queue ensures that only one thread performs read and write operations on the zoom direction state variable at any given time.

[0122] In the Android system, the same thread synchronization effect can be achieved using synchronization locks or atomic variables. The zoom direction and current zoom value returned by the zoom status query interface are also protected by thread synchronization to ensure consistency of the query results. The zoom status query interface is available for other functional modules of the mobile terminal to call, such as displaying the current zoom magnification in the preview interface.

[0123] S500, the zoom command is sent to the gimbal camera so that the gimbal camera performs a zoom operation.

[0124] The mobile terminal sends the generated zoom command to the gimbal camera via the communication module.

[0125] The communication module encapsulates zoom commands into data frames conforming to the gimbal camera's communication protocol and transmits them to the gimbal camera wirelessly. Zoom command transmission is driven by the zoom direction: when the zoom direction is the first zoom direction, zoom-in commands are continuously sent; when the zoom direction is the second zoom direction, zoom-out commands are continuously sent; and when the zoom direction is stopped, a stop command is sent.

[0126] In this embodiment, the wireless communication method is Bluetooth. After the mobile terminal establishes a Bluetooth connection with the gimbal camera, the communication module sends zoom commands to the gimbal camera according to the zoom execution frequency. The communication method can be any of Bluetooth, WiFi, 2.4G wireless, or USB. This embodiment uses Bluetooth, which has the advantages of low power consumption and convenient connection establishment. The substitutability of the communication method is reflected in the following: WiFi transmission can be used in indoor fixed deployment scenarios, with a longer transmission distance; 2.4G wireless transmission can be used in scenarios requiring low latency.

[0127] The communication and transmission process of zoom commands is as follows: The mobile terminal constructs a zoom command data frame, which includes a frame header, command type, speed value, checksum, and frame tail; the communication module sends the data frame to the gimbal camera via Bluetooth; the gimbal camera receives the data frame and parses the command content, drives the stepper motor to control the lens movement, and achieves smooth changes in focal length.

[0128] The gimbal camera response module performs the following operations: The system receives zoom command data frames sent by the mobile terminal, verifies the integrity of the data frames, parses the command type field to determine the zoom direction, determines the zoom speed based on the speed value field, and drives the stepper motor to move the lens in the determined direction and speed.

[0129] For example, when the gimbal camera receives a zoom-in command, it drives the stepper motor to rotate forward, moving the lens towards the telephoto direction and increasing the zoom magnification; when the command is a zoom-out command, it drives the stepper motor to rotate in the reverse direction, moving the lens towards the wide-angle direction and decreasing the zoom magnification. When the command is a stop command, the stepper motor stops rotating, and the current zoom magnification is maintained.

[0130] It should be understood that the continuous transmission of zoom commands is synchronized with the gesture recognition results; that is, when gesture recognition continuously identifies the first gesture type, the mobile terminal continuously sends zoom-in commands, and the gimbal camera continuously executes the operation of increasing the zoom magnification; when gesture recognition continuously identifies the second gesture type, the mobile terminal continuously sends zoom-out commands; when gesture recognition fails to identify a gesture type, the mobile terminal sends a stop command, and the gimbal camera stops zooming. The frequency of zoom command transmission is consistent with the zoom execution frequency, ensuring that each zoom step corresponds to one command transmission, and the zoom execution of the gimbal camera remains synchronized with the zoom steps of the mobile terminal.

[0131] During wireless communication, command loss may occur due to signal interference or excessive distance. The communication module appends a sequence number to each zoom command data frame. After receiving the data frame, the gimbal camera compares the received sequence number with the expected sequence number. When the gimbal camera detects a discontinuity in the sequence number, it determines that a command has been lost. Upon receiving the next data frame, the gimbal camera continues to execute the zoom step in the current direction, ensuring that the zoom process is not interrupted due to the loss of a single frame.

[0132] It should be noted that the communication module does not execute a retransmission mechanism at the sending end because zoom commands are continuously sent at a second frequency. Lost commands are overwritten by commands in the same direction in the next frame after 33 milliseconds. Retransmitting old commands may actually cause the order of command arrival to be inconsistent with the actual zoom direction.

[0133] For example, within the effective range of Bluetooth communication (typically within 10 meters), the single-frame command loss rate is less than 1%, meaning that on average, no more than one frame is lost out of every 100 commands, corresponding to approximately 0.035 times the zoom step loss, which has a negligible impact on the smoothness of the zoom process. When the communication link is interrupted for more than a preset disconnection time (1 second in this embodiment), the mobile terminal determines that the communication is interrupted, automatically stops the zoom step, and displays a communication interruption prompt on the user interface. After the communication is restored, the user can resume zoom control by making a gesture again.

[0134] S600, when the gesture type is not recognized, the step value is linearly decayed from the current value to a preset value and the zoom step is stopped.

[0135] When the gesture recognition process fails to recognize a gesture type, the mobile terminal stops performing zoom stepping. Failure to recognize a gesture type includes the following situations: No hand area was detected in the gesture image; a hand area was detected but no gesture type belonging to the first or second gesture type was identified; and the user moved their hand out of the camera's field of view.

[0136] In the above scenario, the mobile terminal enters a deceleration phase: the step value no longer remains at its current value, but instead linearly decays from the current value to zero as the deceleration duration increases; after the deceleration phase ends, the timer stops triggering the zoom step, the zoom direction state is set to stop, and the current zoom value remains unchanged. In this embodiment, the deceleration duration is 0.3 seconds, meaning that after the gesture disappears, the zoom step smoothly decays from the current step value to zero (i.e., the preset value) within 0.3 seconds, avoiding sudden pauses in the zoom process.

[0137] It should be understood that in this embodiment, the control logic that automatically stops zooming when the gesture disappears and the control logic that continues zooming when the gesture is active complement each other. When gesture recognition continuously outputs the first or second gesture type, zooming continues; when gesture recognition does not output the aforementioned gesture type, zooming automatically stops. This control logic achieves an intuitive mapping where zooming starts when a gesture is present and stops when the gesture disappears, allowing the user to terminate zooming without additional operation.

[0138] In addition, the image stabilization mechanism also applies to zoom-stop scenarios. Occasionally, a single frame recognition failure may cause a brief loss of gesture type. If the duration of this loss is less than the image stabilization confirmation window, zoom-stop will not be triggered, thus avoiding interruption of the zoom process due to single-frame recognition shake. The value of the image stabilization confirmation window is consistent with the preset time window used for zoom direction switching; in this embodiment, it is set to 200 milliseconds.

[0139] For example, when a user holds a palm gesture, if the gesture type is not recognized in a certain frame due to partial occlusion of the hand, and the duration of occlusion is less than 200 milliseconds, the zoom stepping continues without pause; if the duration of occlusion exceeds 200 milliseconds, the zooming enters the deceleration phase and eventually stops.

[0140] If the user makes a gesture again during the deceleration phase, the mobile terminal detects the gesture type and terminates the deceleration phase, entering the acceleration phase. The duration of the acceleration phase is reset to 0. At this time, the zoom step value starts to increase linearly from the current value when deceleration stopped to the preset maximum step value, instead of starting from 0, thus avoiding a significant drop in zoom speed when making rapid and repeated gestures.

[0141] In one implementation, to avoid the deceleration phase being mistakenly terminated due to jitter in single-frame recognition, the mobile terminal terminates the deceleration phase and enters the acceleration phase only after continuously recognizing the type of gesture made again within a preset time window.

[0142] In one implementation, when the gesture type re-identified during the deceleration phase is different from the gesture type before entering the deceleration phase, the mobile terminal stops the deceleration phase and enters the acceleration phase. The zoom step value starts to linearly increase from the current value when the deceleration stops to the preset maximum step value, and the zoom direction is switched to the zoom direction corresponding to the re-identified gesture type.

[0143] Specifically, the zoom step value is a scalar value whose magnitude is independent of the zoom direction. The zoom direction only determines the direction of increase or decrease of the current zoom value: when the zoom direction is the first zoom direction, the current zoom value increases by the step value; when the zoom direction is the second zoom direction, the current zoom value decreases by the step value. Therefore, when a change in gesture type causes a switch in zoom direction, the step value does not need to be reset to zero. Instead, it maintains the current value at the time of deceleration, only switching the direction of increase or decrease of the zoom direction to the direction corresponding to the target direction. The step value continues to increase linearly from this current value to the preset maximum step value.

[0144] Therefore, when users switch gesture types during the deceleration phase (such as switching from a zoom-in gesture to a zoom-out gesture), the zoom speed will not drop to zero and restart due to the change in direction. Instead, it will maintain the current speed and smoothly switch directions, avoiding sudden screen jumps when switching directions and improving the continuity and smoothness of zoom control.

[0145] After zooming stops, the mobile terminal generates a stop command and sends it to the gimbal camera, which then stops driving the stepper motor. The current zoom level remains unchanged after stopping, and zooming resumes from the current zoom level when the user subsequently recognizes the target gesture type. This intermittent operation mechanism allows users to achieve fine-grained zoom control through intermittent gestures.

[0146] For example, a user makes a slap gesture for 1 second and then retracts the gesture: the first 0.5 seconds are an acceleration phase, with the zoom increasing by approximately 0.26x; the next 0.5 seconds are a constant speed phase, with the zoom increasing by approximately 0.53x; after retracting the gesture, a deceleration phase begins, with the zoom increasing by approximately 0.16x, then stopping, with the zoom increasing from 1.0x to approximately 1.95x. After observing the preview and determining that further zooming is needed, the user makes the slap gesture again for 0.5 seconds. This 0.5 seconds is essentially an acceleration phase, with the zoom increasing by approximately 0.26x. Combined with the approximately 0.16x increase during the deceleration phase after retracting the gesture, the zoom continues to increase from 1.95x to approximately 2.37x and stops. After observing again and determining the zoom level is appropriate, the user no longer makes the gesture, and the zoom level remains at approximately 2.37x. During this intermittent operation, the interval between each disappearance and re-making of the gesture can be any length, and the zoom level remains constant during the interval, without automatically resetting over time.

[0147] For example, the user makes a slap gesture (corresponding to the zoom direction) for 1 second and then retracts the gesture, initiating a deceleration phase of zooming. At 0.1 seconds into the deceleration phase, the step value decreases from a preset maximum step value of 0.035 times to approximately 0.023 times. At this point, the user makes a peace sign gesture (corresponding to the zoom-out direction). The mobile terminal recognizes that the gesture type is different from the slap gesture before entering the deceleration phase, terminates the deceleration phase, and the step value resumes linear growth from 0.023 times. Simultaneously, the zoom direction switches to the zoom-out direction, and the current zoom value decreases by a step value of 0.023 times from the current value, increasing to 0.035 times within 0.5 seconds before decreasing at a constant speed. Throughout the entire direction switching process, the zoom speed does not drop to zero, and the image changes smoothly and continuously, avoiding any abrupt jumps during direction switching.

[0148] After zooming stops, the mobile terminal sets the zoom direction state to "stop." The timer stops triggering but is not destroyed. When the user makes another gesture, the gesture recognition process updates the zoom direction state, the timer resumes triggering, and the step value restarts from zero, linearly increasing with the duration of the new gesture type. Zooming continues from the current zoom value. The response time for zoom recovery is equal to the value of the image stabilization confirmation window (200 milliseconds in this embodiment), meaning that zooming begins approximately 200 milliseconds after the user makes a gesture. For example, the zoom magnification is 2.4x when stopped. After the user makes a "palm" gesture again, the zoom continues to increase from 2.4x until the user retracts the gesture or reaches the maximum zoom magnification.

[0149] Example 2 In Example 1, the zoom speed during the constant speed phase is determined by a preset maximum step value and a second frequency, and is independent of the distance between the user and the mobile terminal. In actual use cases, the distance between the user and the mobile terminal is dynamic: when the user is close, the hand occupies a large portion of the screen, and the user can clearly observe the changes in focal length in the preview image, tending towards a faster zoom speed to quickly reach the target focal length; when the user is far away, the hand occupies a small portion of the screen, and the user's observation accuracy of the preview image decreases, tending towards a slower zoom speed to finely adjust the focal length; therefore, a fixed constant zoom speed cannot adapt to these two different control requirements.

[0150] This embodiment introduces a hand area sensing mechanism based on the zoom stepping mechanism of Embodiment 1, so that the preset maximum step value is calculated in real time according to the area of ​​the palm detection frame detected in the gesture recognition process, thereby adaptively adjusting the zoom speed at each stage.

[0151] Specifically, in the hand key point detection step of Example 1, the palm detection model outputs a palm detection box, the width of which is... Height is The units are normalized coordinates (range 0 to 1). The mobile terminal calculates the normalized area of ​​the hand detection bounding box:

[0152] In the formula, The normalized area of ​​the hand detection frame ranges from 0 to 1. The size of the image is related to the distance between the user's hand and the camera: the closer the user is to the camera, the larger the image of the hand. The larger the size; the farther the user is, The smaller.

[0153] Mobile terminals based on palm area Calculate the preset maximum step value in this embodiment. :

[0154] In the formula, The preset maximum step value is the lower limit for small area; in this embodiment, it is set to 0.015 times, which is suitable for fine focusing at long distances. This is the upper limit of the preset maximum step value when the area is large. In this embodiment, it is set to 0.06 times, which is suitable for fast focusing at close range. The lower limit threshold for the area is set to 0.01 in this embodiment, which corresponds to the hand occupying about 1% of the image area, typically at a distance of more than 3 meters. The upper limit threshold for the area is set to 0.08 in this embodiment, which corresponds to the hand occupying about 8% of the image and a typical distance of about 0.5 meters or less. For the truncation function, Limited to Within the range, to prevent extreme area values ​​from causing the preset maximum step value to exceed the limit.

[0155] It should be noted that the maximum step value is preset here. and Linear interpolation is used between the two values, with the interpolation coefficients determined by the normalized position of the palm area within the effective interval. When Time to take ,when Time to take , exist and The step value increases linearly between zero and the latest value during the acceleration phase. Within the dynamic stepping framework of Example 1, the step value still increases linearly from zero to the latest value during the acceleration phase. During the uniform speed phase, maintain as During the deceleration phase, the value decreases linearly from the current value to zero.

[0156] The preset maximum step value is updated at the same frequency as the gesture recognition frequency (i.e., the first frequency). After each gesture recognition is completed, the mobile terminal recalculates the preset maximum step value based on the latest palm area and transmits it to the zoom execution stage. When the zoom execution stage executes the zoom step at the second frequency, it uses the latest preset maximum step value. The preset maximum step value can be adjusted in real time as the user's distance changes during the zoom process.

[0157] To avoid frequent jumps in the preset maximum step value caused by instantaneous hand area jitter, the mobile terminal performs an exponential moving average filter on the hand area of ​​consecutive frames:

[0158] In the formula, The smoothing coefficient is set to 0.3 in this embodiment. The smaller the value, the stronger the filtering effect and the smoother the change in step value, but the less responsive it is to changes in user distance. A larger value results in a more sensitive response, but also weaker suppression of measurement noise. A value of 0.3 strikes a balance between response speed and smoothness.

[0159] For example, a user stands about 2 meters away from the mobile terminal and makes a slapping gesture, the palm area... =0.03, corresponding to a preset maximum step value of The zoom speed during the constant speed phase is times that of the constant speed phase. Times per second. Afterwards, the user moves closer to the mobile terminal to approximately 0.8 meters, increasing the palm area to... =0.06, preset maximum step value updated to The zoom speed increases to times during the constant speed phase. Zoom speed per second. The preset maximum zoom value reflects changes in user distance in real time; zooming accelerates as the user approaches and decelerates as the user moves away, requiring no additional operation.

[0160] Example 3 In Example 1, the step value increases linearly with the duration of the gesture type during the acceleration phase, maintains a preset maximum step value during the constant speed phase, and linearly decays to zero after the gesture disappears. Therefore, based on Example 1, the piecewise function of the zoom step value changing with the duration of the acceleration phase can be expressed as:

[0161] In the formula, The duration of the acceleration phase, in seconds; This is the initial step value for the acceleration phase; The preset acceleration time is set to 0.5 seconds here; The preset maximum step value is set to 0.035 times. The mobile terminal maintenance gesture duration timer starts at the beginning of the acceleration phase. The timer starts counting from 0 and increments by one timer cycle each time it is triggered. ,in (For the second frequency).

[0162] Here The value can be determined in two scenarios: when the gesture type is recognized for the first time, or when zooming stops and restarts. Take zero, and the step value increases linearly from zero to... If deceleration is aborted during the deceleration phase due to the re-identification of the gesture type, Take the step value when deceleration stops The step value continues to increase linearly from the current value to... It does not return to zero when the zoom direction is switched.

[0163] It should be noted that, in When zero is taken, the zoom step value increases linearly from 0 to... Duration .exist With a zoom range of 0.5 seconds and a second frequency of 30 times / second, the acceleration phase consists of 15 steps. The first step has a zoom factor of approximately 0.0023x, and the 15th step increases to 0.035x. At the end of the acceleration phase, the cumulative change in zoom magnification is... =0.035 0.5 / 2 30 = 0.2625 times, which is about half of the change in the same amount of time during the uniform speed phase, thus achieving a gradual start-up.

[0164] Preset acceleration time here The value is determined as follows: when the acceleration time is too short (less than 0.2 seconds), the acceleration process is completed within 6 step cycles, the acceleration effect is not obvious, and the image still has a perceptible jump; when the acceleration time is too long (more than 1 second), the user needs to continuously perform a gesture for more than 1 second to achieve full-speed zoom, resulting in a sluggish operating experience. The 0.5-second acceleration time allows the transition from stationary to full-speed zoom to be completed within 15 step cycles, resulting in a smooth transition and acceptable response speed.

[0165] When the user withdraws their gesture, the mobile device does not immediately stop zooming; instead, it enters a deceleration phase. The duration of this deceleration phase is... In this embodiment, 0.3 seconds is used. During the deceleration phase, the zoom step value linearly decays from the current value to 0.

[0166] In the formula, This refers to the elapsed time after entering the deceleration phase. This is the step value when entering the deceleration phase. After the deceleration phase ends, the mobile terminal stops zooming. With a setting of 0.3 seconds, the deceleration phase includes 9 steps, with the step value ranging from... As the zoom level linearly decreases to near zero, the cumulative change in zoom magnification is approximately... The zoom process ends smoothly, eliminating the sudden braking sensation of the image stopping abruptly.

[0167] If the user makes a gesture again during the deceleration phase, the mobile terminal detects the gesture type, terminates the deceleration phase, and enters the acceleration phase. The duration of the acceleration phase is reset to 0, and the initial step value of the acceleration phase is... Take the step value when deceleration stops At this point, the zoom step value resumes its linear increase from the current value at the time of deceleration termination. Instead of starting from 0, it avoids a significant drop in zoom speed when making rapid, repetitive gestures.

[0168] During the acceleration phase following the deceleration halt, when the re-identified gesture type differs from the gesture type before entering the deceleration phase, the zoom step value linearly increases from its current value at the time of deceleration halt to a preset maximum step value. The change of this step value over time satisfies the following relationship:

[0169] In the formula, The moment when deceleration stops and acceleration resumes. zoom step value, This is the zoom step value when deceleration stops. The duration of the acceleration phase after deceleration stops (starting from 0). hour, That is, when switching zoom direction, the step value remains unchanged; when hour, This means that after a preset acceleration period, the step value increases to a preset maximum step value. The above formula shows that when switching zoom directions, the step value does not need to return to zero, but continues to increase from the current value, ensuring the continuity of zoom speed.

[0170] For example, when a user makes a slapping gesture: the first 0 to 0.5 seconds are the acceleration phase, with the zoom speed increasing linearly from 0 to 1.05x / second, and the zoom magnification increasing from 1.0x to approximately 1.26x; after 0.5 seconds, the zoom speed remains constant at 1.05x / second; at the 2.0-second mark, the user retracts the gesture, and the zoom magnification is approximately 2.84x, entering the deceleration phase, where the zoom speed linearly decreases to 0 within 0.3 seconds, the zoom magnification increases again by approximately 0.16x, and finally stops at approximately 3.0x. The speed curve of the entire zoom process is trapezoidal (acceleration-constant speed-deceleration), with a smooth transition in the image.

[0171] Example 4 The zoom stepping mechanism in Example 1 is based on the deterministic recognition result of gesture type. That is, if a specific gesture type is recognized, the corresponding zoom step is executed; if not recognized, it enters deceleration and stops, without intermediate states. However, in actual operation, the recognition confidence of the gesture recognition model varies across different frames. For example, when the user's gesture is standard and the lighting is sufficient, the confidence is close to 1.0; when the user's gesture is not standard, the hand is partially obscured, or the lighting is insufficient, the confidence may only be 0.5 to 0.7. If the low-confidence frame is still executed with the stepping value determined in Example 1, when the next frame fails to recognize the gesture due to confidence fluctuations, the zoom will enter the deceleration phase, resulting in an unstable zoom process with frequent starts and stops.

[0172] In response, this embodiment introduces a confidence weighting mechanism based on embodiment 1, using the gesture confidence score output by the gesture recognition model as the modulation signal for zoom stepping, so that the strength of zoom stepping is proportional to the degree of certainty of recognition.

[0173] Specifically, the gesture recognition model outputs both the gesture type and the gesture confidence score. The value ranges from 0 to 1. The mobile terminal determines the confidence level based on the gesture. Calculate the weighting coefficients :

[0174] In the formula, The lower confidence level is set to 0.5 in this embodiment. Identification results below this value are considered unreliable, the weighting coefficient is 0, and zoom stepping is not performed. The upper limit of confidence is set to 0.85 in this embodiment. Identification results with a value higher than this are considered reliable, and the weighting coefficient is 1. exist and When the weighting coefficients are between 0 and 1, they are linearly interpolated.

[0175] Here =0.5 because when the confidence level is below 0.5, the model's judgment of the gesture type is close to random, and the risk of performing zoom step is greater than the benefit at this time; The value of 0.85 is based on the typical output range of the model under standard gestures. A value higher than 0.85 indicates that the model clearly judges the gesture type. The linear interpolation interval allows the zoom step to be gradually adjusted in the uncertain region, so that it will not cause a full step impact due to low confidence frames, nor will it completely lose zoom power when the confidence is slightly lower than the threshold.

[0176] To avoid frequent jumps in weighting coefficients due to inter-frame confidence fluctuations, mobile terminals perform weighting on consecutive frames. Perform first-order low-pass filtering on the value:

[0177] In the formula, The filter coefficient is 0.4 in this embodiment. The smaller the value, the stronger the filtering effect and the smoother the step value, but the less responsive it is to changes in confidence level. A value of 0.4 strikes a balance between smoothness and responsiveness.

[0178] When performing zoom stepping, the current step value determined by the gesture duration in Example 1 is used. Multiply by the smoothed weighting coefficients to obtain the actual step value. The weighting coefficients are updated after each gesture recognition is completed. The zoom execution phase uses the latest weighting coefficients when performing zoom steps at the second frequency.

[0179] For example, when the user makes a slapping gesture, the confidence scores for five consecutive frames are 0.92, 0.88, 0.71, 0.83, and 0.91, respectively: the confidence scores for the first and second frames are higher than... The weighting coefficient is 1.0, and the actual step value is equal to the current theoretical step value; the confidence level of the 3rd frame is 0.71, and the weighting coefficient is (0.71-0.5) / (0.85-0.5)=0.60. After low-pass filtering... The actual step value decreased to 0.84 times the current theoretical step value; the confidence level of the 4th frame was 0.83, and the weighting coefficient was (0.83-0.5) / 0.35=0.94. The confidence level of frame 5 is 0.91, which is higher than... The weighting factor is 1.0. In the above process, low-confidence frames cause the actual step value to temporarily decrease, rather than being executed in full steps or suddenly entering deceleration. Low-pass filtering makes the change in step value inertia, avoiding zoom jumps caused by single-frame confidence fluctuations.

[0180] When user gestures are not standard or lighting is insufficient, causing confidence scores to remain low for multiple frames. hour, The value gradually decays to 0 after multi-frame low-pass filtering, and the actual step value naturally decreases to 0, resulting in no sudden pause during zooming. Compared to the stop mechanism based on the image stabilization time window in Example 1, the confidence weighting mechanism in this example provides a smoother transition in the unstable region of gesture recognition.

[0181] Example 5 The zoom control in the aforementioned embodiments all directly controls the zoom magnification, requiring users to determine the appropriate zoom magnification themselves. However, in scenarios such as solo video shooting and live streaming, the user's real need is to maintain a suitable compositional proportion for the subject in the frame: when the distance between the subject and the lens changes, if the zoom magnification is fixed, the proportion of the subject in the frame will change with the distance, requiring the user to frequently make corrections.

[0182] In this embodiment, the viewfinder image transmitted back by the gimbal camera is used as feedback, and the size of the subject in the image is used as the controlled variable. After the user selects the target subject with a one-time gesture, the mobile terminal automatically controls the zoom in a closed loop to keep the size of the subject in the image at a preset ratio. When the subject moves away, it automatically zooms in to compensate, and when the subject moves closer, it automatically zooms out to compensate.

[0183] This embodiment is applicable to deployments where a mobile terminal and a gimbal camera have the capability to transmit viewfinder images back to each other. Specifically, the gimbal camera compresses its viewfinder image and periodically transmits it back to the mobile terminal. The mobile terminal displays the transmitted image locally and performs subject detection on the transmitted image. Subject detection can use a face detection model or a pedestrian detection model; this embodiment uses a face detection model with a detection period of 100 milliseconds.

[0184] When the gesture recognition module recognizes a preset lock gesture, the mobile terminal enters hold mode. The preset lock gesture is a one-time trigger gesture, which is the same as the triggering method of the aforementioned one-time function gestures. For example, it can be configured as a double slap gesture or a "666" gesture lasting more than 1 second. After recognition, it only triggers a mode switch once and does not require the user to hold the gesture continuously.

[0185] After entering hold mode, the mobile terminal determines the target subject in the current transmitted image: when only one face is detected in the transmitted image, the detection box corresponding to that face is determined as the target subject; when multiple faces are detected, the user's facial features stored in the mobile terminal are matched with each face detection box, and the face with the highest matching degree is determined as the target subject, thus avoiding locking the wrong object in a multi-person scenario.

[0186] Simultaneously, the mobile terminal records the desired size of the target subject. In this embodiment, the desired size is defined as the ratio of the subject detection frame height to the screen height. express, It can be configured within the range of 0.3 to 0.7, with a default value of 0.5, meaning the subject's height in the image occupies approximately half the image height. Correspondingly, the desired pixel height... ,in This represents the pixel height of the transmitted image.

[0187] After the hold mode is activated, the mobile terminal acquires the current size of the target subject according to the detection cycle and calculates the composition error. Let the height of the currently detected target subject detection frame be denoted as... The composition error is:

[0188] A composition error greater than zero indicates that the subject is too small in the image and needs to be enlarged; a composition error less than zero indicates that the subject is too large in the image and needs to be reduced.

[0189] To suppress single-frame detection jitter, the mobile terminal performs first-order exponential smoothing on the detection box height:

[0190] In the formula, As a smoothing coefficient, it is set to 0.6 in this embodiment to give the change in the main body size a moderate inertia and eliminate the size fluctuation of single-frame detection.

[0191] The mobile terminal then determines the zoom rate based on the composition error:

[0192] In the formula, For reference zoom rate, 0.8x / second is used in this embodiment, which is the typical upper limit of smooth zoom rate for gimbal cameras; A positive value indicates the direction of magnification, and a negative value indicates the direction of reduction. The zoom rate is directly proportional to the relative composition error; the greater the composition error, the faster the zoom, and the smaller the error, the slower the zoom.

[0193] The zoom increments are still driven by the second frequency timer, consistent with Example 1, at 30 times / second. The step value for each zoom increment is... ,in The second frequency, with its sign determining the zoom direction, is used by the mobile terminal to generate zoom-in or zoom-out commands and send them to the gimbal camera.

[0194] To avoid frequent start-stop cycles of the lens motor when the subject size fluctuates around the target value, a dead zone is set in the mobile terminal: when the absolute value of the composition error is less than a preset error threshold, the composition is deemed to meet the requirements, and zoom increments are stopped. In this embodiment, the preset error threshold is set to 5% of the desired pixel height, i.e. The dead zone setting stops when the subject size deviation is within 5%. This value is based on the fact that the human eye can hardly perceive the compositional change when the subject size deviation is within 5%. Continuing to drive the lens motor at this time will only cause meaningless power consumption and wear. The dead zone setting serves both image stabilization and motor protection.

[0195] If the target subject is not detected in the returned image for a consecutive preset number of frames (e.g., the target subject turns around, is obscured, or walks out of the frame), the mobile terminal stops performing zoom steps and maintains the current zoom level to avoid blindly zooming when the subject is lost. When the target subject is detected again, closed-loop control resumes. The preset number of frames here is 20, corresponding to approximately 2 seconds of loss confirmation time. In hold mode, if the gesture recognition module recognizes a preset exit gesture or the aforementioned directional gesture, the mobile terminal exits hold mode and returns manual control.

[0196] For example, the height of the returned image is 1080 pixels, and the expected percentage is... If the value is 0.5, the expected pixel height is 540 pixels. At time t0, the subject's height in the frame is 540 pixels, the composition error is zero, zooming stops, and the image stabilizes. The subject then moves backward, and its height is detected to have dropped to 378 pixels, approximately 35% of the frame. The composition error is 162 pixels, with a relative error of 0.3. The zoom rate is determined to be 0.3 x 0.8 = 0.24x / second, corresponding to a step value of approximately 0.008x / step. The mobile terminal sends a zoom command to the gimbal camera. The subject continues to move backward to a height of 324 pixels, the relative error increases to 0.4, and the zoom rate increases to 0.32x / second. The subject stops moving backward and slowly approaches, its height gradually increases, the composition error gradually decreases, and the zoom rate decreases accordingly. When the subject's height rises above 513 pixels, the composition error is less than 27 pixels, entering a dead zone, zooming stops, and the subject size remains within a tolerance of approximately 5% of the expected percentage. Throughout the process, the user only specified the subject with a one-time gesture at the beginning and did not maintain any gesture continuously; the zoom was completed automatically.

[0197] This application also provides a gesture recognition-based intelligent zoom control system for a remote gimbal camera. For example... Figure 4 As shown, the system includes: The gesture acquisition module is used to acquire gesture image sequences. It is connected to the front-facing camera of the mobile terminal to acquire user gesture images and output gesture image sequences.

[0198] The image preprocessing module is used to perform rotation correction on the gesture images in the gesture image sequence according to the device orientation, and to perform preprocessing such as rotation correction and size scaling on the acquired video frames. The processed gesture images are then output to the gesture recognition module.

[0199] The image preprocessing module obtains the current device orientation of the mobile terminal, which includes four states: portrait orientation, portrait orientation upside down, landscape orientation with left turn, and landscape orientation with right turn.

[0200] The mobile terminal obtains the current device orientation through its built-in orientation sensor or system interface. The image preprocessing module determines the rotation angle based on the device orientation: 0 degrees for portrait orientation, 180 degrees for inverted orientation, 90 degrees for leftward rotation in landscape orientation, and 270 degrees for rightward rotation in landscape orientation. The image preprocessing module performs an affine rotation transformation on the gesture image, rotating it to a uniform vertical direction by the determined rotation angle. The orientation of the hand region in the rotated gesture image is consistent with the orientation of the gesture recognition model's training data, avoiding deviations in hand keypoint positioning caused by changes in device orientation.

[0201] For example, when a user uses a mobile terminal in landscape mode, the gesture image captured by the front-facing camera is rotated 90 degrees relative to the portrait mode. Without rotation correction, the hand detection model may not be able to correctly locate the hand area. After the image preprocessing module rotates the gesture image by 90 degrees, the hand area is restored to the vertical direction, and the hand detection model can detect it normally. The image preprocessing module also performs size scaling on the rotated gesture image, scaling it to the input size requirements of the hand detection model. The aspect ratio remains unchanged during the scaling process, and any insufficient parts are padded with padding values.

[0202] The gesture recognition module is used to perform gesture recognition on the gesture image sequence at a first frequency to obtain the gesture type, which is either a first gesture type or a second gesture type.

[0203] The gesture recognition module includes a gesture detection unit and a gesture recognition unit. The gesture detection unit detects the hand position and key points of the hand through a gesture recognition model. The gesture recognition unit determines the opening and closing state of each finger based on the key points of the hand and identifies the gesture type based on the combination of the opening and closing states of each finger.

[0204] The gesture recognition module is also used to distinguish gesture types based on a distance threshold between the fingertips of the thumb and index finger when the combination of open and closed states is the same.

[0205] The zoom control module is used to determine the zoom direction based on the gesture type. During the period when the gesture type is recognized, it continuously executes zoom steps at a second frequency and generates zoom commands based on the zoom direction. When the gesture type is not recognized, it causes the step value to decay linearly from the current value to zero and stops executing the zoom steps.

[0206] The zoom direction can be either a first zoom direction or a second zoom direction, and the second frequency is independent of the first frequency. The zoom control module includes a direction determination unit, a stepping execution unit, and a command generation unit. The direction determination unit determines the target zoom direction based on the gesture type, and switches the zoom direction after confirmation within a preset time window when the gesture type changes; the step execution unit is used to time the duration of the gesture type, and continuously executes zoom steps at a second frequency through a timer, so that the step value linearly increases from zero to the preset maximum step value when the duration is less than the preset acceleration duration, and remains at the preset maximum step value after the duration reaches the preset acceleration duration, and linearly decays the step value from the current value to zero when no gesture type is recognized, and stops zoom steps when the zoom magnification reaches the maximum zoom magnification or minimum zoom magnification; the instruction generation unit generates zoom-in instructions, zoom-out instructions or stop instructions based on the current zoom direction.

[0207] The communication module is used to send zoom commands to the gimbal camera, establish a connection with the gimbal camera via wireless communication, encapsulate the zoom commands into data frames that conform to the gimbal camera's communication protocol, and send them.

[0208] The gimbal camera is used to receive zoom commands and execute zoom operations, and includes a communication interface, a command parsing unit, a stepper motor, and a lens zoom mechanism. The communication interface receives zoom command data frames sent by the mobile terminal, the command parsing unit parses the command type field to determine the zoom direction, and the stepper motor drives the lens zoom mechanism to move in the determined direction to adjust the focal length.

[0209] In this system embodiment, the functions of each module correspond one-to-one with the execution processes of each step in the aforementioned method embodiment. The gesture acquisition module 100 corresponds to S100, the image preprocessing module 200 corresponds to the rotation correction and scaling of the gesture image, and the gesture recognition module corresponds to S200. Specifically, the gesture detection module 300 detects the hand position and key hand points, and the gesture recognition module 400 determines the finger opening / closing state and gesture type based on the key hand points. The zoom control module corresponds to S300, S400, and S600. Specifically, the gesture state management module 500 manages the current gesture state and determines whether the gesture persists, the zoom command generation module 600 generates corresponding zoom commands based on the gesture type, and the zoom execution module 700 executes zoom steps at a second frequency using a timer and dynamically adjusts the step value based on the gesture duration. The communication module 800 and the gimbal camera 900 correspond to S500. The gesture frequency statistics module 1000 counts the call frequency of gesture recognition as a reference for the zoom execution frequency.

[0210] Each module can be implemented by the mobile terminal's processor executing computer program instructions stored in its memory, or it can be implemented by dedicated hardware circuitry. It should be noted that the division of modules in this system embodiment is merely an exemplary logical functional division; in actual implementation, the functions of multiple modules can be merged or further split as needed.

[0211] In another embodiment, the gesture control system provided in this application also supports mapping different gesture types to other control functions of the gimbal camera.

[0212] For example, a "like" gesture can be mapped to a pan / tilt camera landscape / portrait switching function. When the gesture recognition module detects a "like" gesture with only the thumb open and pointing upwards, the zoom control module does not execute zoom steps but instead generates a landscape / portrait switching command and sends it to the pan / tilt camera via the communication module. Upon receiving this command, the pan / tilt camera switches the orientation of the captured image. A "666" gesture can be mapped to a photo or video recording function. When the gesture recognition module detects a "666" gesture with the thumb and little finger open and the other fingers closed, it generates a photo or video recording trigger command and sends it to the pan / tilt camera. A double-hand slap gesture can be mapped to a photo and video recording mode switching function. When the gesture recognition module detects two hands simultaneously, both in a slap gesture, it generates a mode switching command.

[0213] The aforementioned non-zoom gesture control functions adopt a one-time trigger mode: after the gesture recognition module recognizes the target gesture, it triggers the corresponding function only once, unlike zoom control which executes continuously. To avoid repeated triggering of one-time functions due to gesture shakiness, an independent anti-shake time window is set for one-time trigger functions. For example, the anti-shake time window for the "like" gesture is 400 milliseconds, and the anti-shake time window for the "666" gesture and the double slap gesture is 600 milliseconds. The anti-shake mechanism is implemented as follows: the mobile terminal records the last trigger time of each one-time function. When the gesture recognition module recognizes the corresponding gesture again, it checks whether the interval between the current time and the last trigger time is greater than the anti-shake time window. If it is greater, the function is executed and the trigger time is updated; if it is less than or equal to, the recognition result is ignored. The anti-shake time window for one-time functions is usually larger than the preset time window for zoom direction switching because the cost of accidental touches of one-time functions is higher. Accidental zoom direction switching only causes a brief change in direction, which can be corrected by switching again; however, accidental touches when taking photos will produce unwanted photos, and accidental switching between portrait and landscape modes will interrupt the continuity of the shooting scene.

[0214] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware.

[0215] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for intelligent zoom control of a remote gimbal camera based on gesture recognition, applied to a mobile terminal, wherein the mobile terminal is communicatively connected to the gimbal camera, characterized in that... The method includes: Obtain the gesture image sequence; The gesture image sequence is subjected to gesture recognition at a first frequency through a gesture recognition model to obtain the gesture type, which is either a first gesture type or a second gesture type. The zoom direction is determined based on the gesture type, and the zoom direction is either a first zoom direction or a second zoom direction. During the recognition of the gesture type, zoom steps are continuously executed at a second frequency, and zoom commands are generated according to the zoom direction. The second frequency is independent of the first frequency. The step value of the zoom step is dynamically adjusted according to the duration of the gesture type. When the duration is less than a preset acceleration duration, the step value linearly increases from zero to a preset maximum step value. After the duration reaches the preset acceleration duration, the step value remains at the preset maximum step value. The first frequency is lower than the second frequency. The zoom command is sent to the gimbal camera to cause the gimbal camera to perform a zoom operation; When the gesture type is not recognized, the step value decreases linearly from the current value to a preset value, and the zoom stepping stops.

2. The method according to claim 1, characterized in that, During the process of the step value linearly decreasing from the current value to the preset value, when the gesture type is re-identified, the decreasing is stopped, and the step value linearly increases again from the current value during the decreasing process to the preset maximum step value.

3. The method according to claim 2, characterized in that, During the process of the step value linearly decaying from the current value to the preset value, when the re-identified gesture type is different from the gesture type before entering the deceleration phase, the step value linearly increases again from the current value during the decay process to the preset maximum step value, and at the same time, the zoom direction is switched to the zoom direction corresponding to the re-identified gesture type.

4. The method according to claim 1, characterized in that, When the gesture type is not recognized, the step value linearly decreases from the current value to the preset value within a preset deceleration time, where the preset deceleration time is less than the preset acceleration time.

5. The method according to claim 1, characterized in that, The first gesture type corresponds to a gesture with all fingers spread, and the second gesture type corresponds to a gesture with some fingers spread.

6. The method according to claim 5, characterized in that, The step of performing gesture recognition on the gesture image sequence at a first frequency includes: Hand key points are detected based on the gesture images in the gesture image sequence; The opening and closing state of each finger is determined based on the key points of the hand. The gesture type is identified based on the combination of the opening and closing states of each finger; Where the combinations of the open and closed states are the same, the gesture type is distinguished according to the distance threshold between the fingertips of the thumb and index finger.

7. The method according to claim 6, characterized in that, Determining the opening and closing state of each finger based on the key points of the hand includes: Determine the angle between the direction vectors of adjacent phalanges based on the key points of adjacent phalanges of each finger; The angle between the direction vectors is compared with a preset angle threshold. If the angle between the direction vectors is less than the preset angle threshold, the corresponding finger is determined to be in an open state.

8. The method according to claim 1, characterized in that, Determining the zoom direction based on the gesture type includes: When the gesture type changes, after continuously recognizing the changed gesture type within a preset time window, the zoom direction is switched to the zoom direction corresponding to the changed gesture type. Wherein, the first zoom direction corresponds to increasing the zoom magnification, and the second zoom direction corresponds to decreasing the zoom magnification.

9. The method according to claim 1, characterized in that, When the zoom level corresponding to the zoom step reaches the maximum or minimum zoom level, the zoom step is stopped.

10. A remote gimbal camera intelligent zoom control system based on gesture recognition, characterized in that, include: Gesture acquisition module, used to acquire gesture image sequences; A gesture recognition module is used to perform gesture recognition on the gesture image sequence at a first frequency to obtain a gesture type, wherein the gesture type is a first gesture type or a second gesture type. A zoom control module is configured to determine the zoom direction based on the gesture type, continuously execute zoom steps at a second frequency and generate zoom commands based on the zoom direction while the gesture type is recognized, and linearly decay the step value from the current value to a preset value and stop executing the zoom steps when the gesture type is not recognized; wherein, the zoom step value is dynamically adjusted according to the duration of the gesture type, when the duration is less than a preset acceleration duration, the step value linearly increases from zero to a preset maximum step value, and after the duration reaches the preset acceleration duration, the step value remains at the preset maximum step value; the zoom direction is either a first zoom direction or a second zoom direction, and the second frequency is independent of the first frequency; the first frequency is lower than the second frequency. The communication module is used to send the zoom command to the gimbal camera; A gimbal camera is used to receive the zoom command and perform the zoom operation.

Images