Gesture interaction method, apparatus, computing device, and storage medium

Abstract

The application relates to a gesture interaction method and device, a computing device and a storage medium, and belongs to the technical field of human-computer interaction. The method comprises the following steps: obtaining a first image containing an operator from an original video stream collected by a camera, recognizing a human body reference length of the operator, and measuring a pixel length of the human body reference length in the first image; determining a second image containing a hand of the operator based on the first image, and obtaining coordinates of hand key points in the second image; determining a gesture interaction parameter according to the pixel length; wherein the gesture interaction parameter is adjusted according to the change of the pixel length; determining an intention of an interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameter, so as to generate a control instruction of a gesture interaction operation. The technical scheme of the application can make the same physical amplitude of hand movement produce consistent screen operation effects at different distances.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a gesture interaction method, device, computing device and storage medium. Background Technology

[0002] With the rapid development of display technology, ultra-large display terminals (such as smart conference screens, interactive walls in digital exhibition halls, and large commercial advertising screens) have been widely used in various public and commercial spaces. Unlike traditional personal computers or mobile devices, these large-screen devices are usually placed in open areas, and users often need to perform contactless operation from a distance of 3 to 5 meters or even further. Currently, vision-based gesture interaction technology has become the mainstream solution for achieving this kind of long-distance control. Its basic principle is to use a camera to collect video streams of user movements, identify the coordinates of key points on the hand, and map them to cursor movement or operation commands on the screen.

[0003] However, in actual long-distance interaction scenarios, existing gesture control solutions suffer from poor user experience.

[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] In view of this, this application provides a gesture interaction method, apparatus, computing device, and storage medium to solve at least one problem existing in the background art.

[0006] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a gesture interaction method, the method comprising:

[0008] From the raw video stream captured by the camera, a first image containing the operator is obtained, and the human body reference length of the operator is identified. The pixel length of the human body reference length in the first image is measured. The human body reference length is a linear distance with fixed physical dimensions on the operator's body.

[0009] Based on the first image, a second image containing the operator's hand is determined, and the coordinates of key points of the hand in the second image are obtained;

[0010] Gesture interaction parameters are determined based on the pixel length; wherein, the gesture interaction parameters are adjusted as the pixel length changes: when the operator moves away from the camera and the pixel length decreases, the gesture interaction parameters are increased; when the operator moves closer to the camera and the pixel length increases, the gesture interaction parameters are decreased.

[0011] Based on the coordinates of the hand key points in the second image and the gesture interaction parameters, the intention of the interaction operation is determined to generate control instructions for the gesture interaction operation.

[0012] In one alternative embodiment, the human body reference length is the operator's shoulder width;

[0013] Measuring the pixel length of the human body reference length in the first image includes: identifying the coordinates of the operator's left and right shoulder peaks in the first image, and calculating the pixel distance between the two points as the pixel length.

[0014] In one optional implementation, the gesture interaction parameter is a gesture mapping ratio;

[0015] The step of determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, in order to generate control instructions for the gesture interaction operation, includes:

[0016] The coordinates of the key hand points in the second image are converted into operation positions on the screen using the gesture mapping ratio.

[0017] The intention of the interactive operation is determined based on the operation position on the screen, so as to generate control instructions for the gesture interaction operation.

[0018] In one optional implementation, the gesture interaction parameter is a gesture trigger threshold;

[0019] The step of determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, in order to generate control instructions for the gesture interaction operation, includes:

[0020] Calculate the hand posture features or hand movement distance based on the coordinates of the key hand points in the second image;

[0021] The hand posture features or hand movement distance are compared with the gesture trigger threshold. When the gesture trigger threshold is exceeded, it is identified as a valid gesture action, and a corresponding control command for gesture interaction is generated.

[0022] In an optional implementation, determining the gesture interaction parameters based on the pixel length includes:

[0023] Establish the correspondence between the physical area corresponding to the operator's body reference length and the screen display area;

[0024] This allows the hand to move within the human body's baseline length range, corresponding to the operation displacement of the entire screen area or a portion of the screen area.

[0025] In an optional implementation, obtaining the coordinates of the hand key points in the second image includes:

[0026] Identify the operator's hand orientation and determine whether the operator's hand is left-handed or right-handed;

[0027] Calculate the coordinates of the hand root in the second image, and establish a local coordinate system for the hand using the hand root as an anchor point;

[0028] Based on the aforementioned local hand coordinate system, the coordinates of other key points of the hand, excluding the base of the hand, are extracted.

[0029] In an optional implementation, obtaining the coordinates of the hand key points in the second image includes a jump compensation step:

[0030] Get the original coordinates of the hand in the current frame;

[0031] The predicted hand coordinates of the current frame are calculated using a prediction algorithm based on the motion trends of the previous N frames; where N is an integer greater than or equal to 2.

[0032] The original hand coordinates are compared with the predicted hand coordinates. If the deviation between the two exceeds a preset human motion threshold, the current frame is determined to be abnormal, and the predicted hand coordinates are used as the coordinates of the hand key points in the second image. If the deviation is within the normal range, the original hand coordinates are used, and the internal parameters of the prediction algorithm are updated using the original hand coordinates.

[0033] In an optional implementation, after determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters to generate control instructions for the gesture interaction operation, a smoothing process step is further included:

[0034] The adaptive filter cutoff frequency is dynamically adjusted based on the movement speed of the key hand points: when the movement speed is greater than a preset speed threshold, the filter cutoff frequency is increased to reduce signal delay; when the movement speed is less than or equal to the preset speed threshold, the filter cutoff frequency is decreased to suppress signal jitter.

[0035] In an optional implementation, when generating the control command for the gesture interaction operation, a step to prevent accidental touches is further included:

[0036] Monitor changes in the state of hand gestures, and enter a cooldown period after a valid trigger command is detected;

[0037] During the cooling period, subsequent identical trigger signals are ignored to prevent repeated execution of commands due to hand tremors.

[0038] In an alternative implementation, before determining the intent of the interaction based on the coordinates of the hand keypoints in the second image and the gesture interaction parameters, an orientation calibration step is further included:

[0039] Identify the operator's body orientation angle;

[0040] Based on the body's orientation angle, adjust the starting point or direction of the operation area on the screen display area so that when the operator stands sideways, the direction of hand movement is consistent with the operation direction on the screen display area.

[0041] The determination of the intent of the interactive operation specifically includes: determining the intent of the interactive operation based on the adjusted starting point or direction of the operation area.

[0042] In an optional implementation, before determining the intent of the interaction based on the coordinates of the hand key points in the second image and the gesture interaction parameters, a gaze calibration step is further included:

[0043] Identify the operator's eye movement trajectory;

[0044] By combining the human eye gaze trajectory with the human body reference length, a two-layer calibration mechanism is constructed, wherein: the first layer determines the overall mapping ratio based on the human body reference length, and the second layer determines the operation area based on the human eye gaze trajectory;

[0045] The determination of the intent of the interactive operation specifically includes: determining the intent of the interactive operation based on the overall mapping ratio and operation area determined by the dual-layer calibration mechanism.

[0046] In an alternative implementation, determining a second image containing the operator's hand based on the first image includes an image enhancement step:

[0047] Extract a third image containing the hand from the first image;

[0048] The third image is subjected to detail magnification and enhancement processing to obtain an enhanced image with higher clarity than the original video stream, and the enhanced image is used as the second image;

[0049] Based on the second image, key points of the hand are identified.

[0050] In one alternative implementation, the detail magnification and enhancement processing includes:

[0051] The pixels of the third image are reconstructed using a weighted interpolation algorithm;

[0052] A window function is used to limit the range of neighboring pixels involved in the calculation, and smoothing weights are assigned to pixels at different locations to suppress artifacts at image edges.

[0053] In an alternative implementation, before acquiring the first image containing the operator from the raw video stream captured from the camera, an operator locking step is further included:

[0054] Track multiple target objects in the original video stream and assign them unique identifiers;

[0055] Calculate the overlap between each target object and the preset interaction area, the dwell time, and the human body orientation score to obtain the interaction intent score;

[0056] Select the target object with the highest interaction intent score as the operator.

[0057] In an optional implementation, obtaining the coordinates of key hand points in the second image further includes a hand orientation correction step:

[0058] Obtain the angle between the vertical axis of the palm and the vertical axis of the second image; wherein, the vertical axis of the palm is the direction extending from the wrist to the fingers;

[0059] The image region containing the hand is rotated and corrected according to the included angle, so that the longitudinal direction of the palm in the corrected hand image is consistent with the preset standard direction, so as to accurately extract the key points of the hand.

[0060] In one optional embodiment, the key points of the hand include the wrist, palm, and each finger joint;

[0061] When determining the intent of the interactive operation to generate control instructions for the gesture interactive operation, at least one of the following hand state features is also utilized: the degree of finger flexion, used to characterize the opening and closing state of the fingers.

[0062] The direction the palm faces is used to indicate the spatial orientation of the hand;

[0063] The number of fingers extended is used to represent the number of fingers that can be straightened.

[0064] And the relative instantaneous velocity of each finger joint, used to characterize the speed of hand movement.

[0065] In one optional implementation, the prediction algorithm is a Kalman filter algorithm;

[0066] The step of using a prediction algorithm to calculate the predicted hand coordinates for the current frame based on the motion trends of the previous N frames includes:

[0067] Based on the average velocity and acceleration of hand movement in historical frames, the global displacement of the hand is calculated in time series to obtain the predicted coordinates of the hand in the current frame.

[0068] In one alternative implementation, the adaptive filter is a frequency conversion low-pass filter.

[0069] In one alternative implementation, the cooling time is 300 milliseconds to 800 milliseconds.

[0070] Secondly, embodiments of this application provide a gesture interaction device, the device comprising:

[0071] The measurement module is used to acquire a first image containing the operator from the raw video stream captured by the camera, identify the operator's human body reference length, and measure the pixel length of the human body reference length in the first image; wherein, the human body reference length is a linear distance with fixed physical dimensions on the operator's body;

[0072] The acquisition module is used to determine a second image containing the operator's hand based on the first image, and to acquire the coordinates of key points of the hand in the second image;

[0073] The first determining module is used to determine gesture interaction parameters based on the pixel length; wherein the gesture interaction parameters are adjusted as the pixel length changes: when the operator moves away from the camera and the pixel length decreases, the gesture interaction parameters are increased; when the operator moves closer to the camera and the pixel length increases, the gesture interaction parameters are decreased.

[0074] The second determining module is used to determine the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, so as to generate control instructions for the gesture interaction operation.

[0075] Thirdly, embodiments of this application provide a computing device, the computing device comprising: a storage component, a communication bus, and a processing component, wherein:

[0076] The storage component is used to store gesture interaction method programs;

[0077] The communication bus is used to enable communication between the storage component and the processing component;

[0078] The processing unit is used to execute the gesture interaction method program to implement the steps of any of the methods described above.

[0079] Fourthly, embodiments of this application provide a computer-readable storage medium storing an executable program, which, when executed by a processor, implements the steps of any of the methods described above.

[0080] The gesture interaction method, apparatus, computing device, and storage medium provided in this application include: acquiring a first image containing an operator from an original video stream captured by a camera, identifying the operator's human body reference length, and measuring the pixel length of the human body reference length in the first image; determining a second image containing the operator's hand based on the first image, and acquiring the coordinates of key points of the hand in the second image; determining gesture interaction parameters based on the pixel length; wherein the gesture interaction parameters are adjusted as the pixel length changes; and determining the intention of the interaction operation based on the coordinates of the key points of the hand in the second image and the gesture interaction parameters to generate control instructions for the gesture interaction operation. It can be seen that the gesture interaction method, apparatus, computing device, and storage medium in this application dynamically adjust gesture interaction parameters by measuring the pixel length of the human body reference length in the image. Since the physical dimensions of the human body reference length are fixed, its pixel length can reflect changes in the imaging ratio in real time. Adjusting the parameters accordingly can automatically compensate for fluctuations in the mapping ratio caused by changes in distance, ensuring that hand movements of the same physical amplitude produce consistent screen operation effects at different distances.

[0081] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0082] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0083] Figure 1 A flowchart illustrating the gesture interaction method provided in Embodiment 1 of this application;

[0084] Figure 2 A detailed flowchart illustrating the gesture interaction method provided in Embodiment 1 of this application;

[0085] Figure 3 This is a schematic diagram of the structure of the gesture interaction device provided in Embodiment 2 of this application;

[0086] Figure 4 This is a schematic diagram of the structure of the computing device provided in Embodiment 3 of this application.

[0087] Explanation of reference numerals in the attached figures:

[0088] 30. Gesture interaction device; 31. Measurement module; 32. Acquisition module; 33. First determination module; 34. Second determination module; 50. Computing device; 51. Storage component; 52. Communication bus; 53. Processing component; 54. Input device; 55. Output device; 56. External communication interface. Detailed Implementation

[0089] To make the technical solutions and beneficial effects of this application more obvious and understandable, the technical solutions in the embodiments of this application are clearly and completely described below by listing specific embodiments. Obviously, the embodiments of this application are not exhaustive, and the described embodiments are only some embodiments of this application, not all embodiments.

[0090] The exemplary embodiments disclosed in this application will now be described in more detail with reference to the accompanying drawings, providing detailed structures and steps to illustrate the technical solution of this application. Note that the drawings are not necessarily drawn to scale, and local features may be enlarged or reduced to more clearly show the details of the local features.

[0091] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for the purpose of describing particular embodiments only and should not be construed as limiting the technical solutions of this application.

[0092] The following description provides numerous specific details to offer a more thorough understanding of this application. However, it will be apparent to those skilled in the art that this application can be practiced without one or more of these details. To clearly define the inventive concept of this application and avoid confusion with its content, technical features well-known in the art and conventionally understood by those skilled in the art are not elaborated upon. Specifically, this document does not fully list all features of actual embodiments, nor does it provide a detailed description of well-known functions and structures.

[0093] The inventors of this application discovered during research and development that existing gesture control solutions suffer from poor user experience in actual long-distance interaction scenarios for the following reasons: Due to the perspective principle of optical imaging, when a user moves within a large depth range, the pixel displacement of the same hand movement in the image changes with distance, leading to unstable operation sensitivity; simultaneously, it is difficult for users to maintain a standard posture facing the screen consistently during actual operation, and changes in body orientation and the blurring of hand features at long distances can easily cause deviations in control direction or jitter in the operation signal. This makes it difficult for users to obtain a consistent, accurate, and natural interaction experience at long distances, often requiring repeated adjustments to their position or the range of motion to complete the expected operation.

[0094] Therefore, through further research and development, the inventors proposed the following technical solution.

[0095] Example 1

[0096] This application provides a gesture interaction method. The method can be implemented by a computer, which can be a computing device configured with a processor. The processor can be a general-purpose processor, such as a CPU; an integrated system, such as a system-on-a-chip (SoC); an embedded control core, such as a microcontroller unit (MCU); a dedicated signal processing unit, such as a digital signal processor (DSP); a graphics rendering core, such as a graphics processing unit (GPU); a programmable logic device, such as an application-specific integrated circuit (ASIC); a field-programmable gate array (FPGA); other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components, etc.

[0097] Without limitation, this solution does not rely on a specific hardware architecture and can be flexibly deployed on cloud servers or local terminals to adapt to different computing power scenarios. For example, using GPUs or ASICs can accelerate the deep learning inference process and meet the stringent real-time requirements of long-distance interaction; using MCUs is suitable for low-power embedded scenarios.

[0098] In a non-limiting sense, embodiments of this application may be used for edge computing devices equipped with GPUs, which utilize the parallel computing capabilities of graphics processors to accelerate the decoding of video streams and neural network inference, thereby controlling end-to-end interaction latency to the millisecond level and ensuring instant response to user operations.

[0099] The technical solution of this application will be described in detail below with reference to the accompanying drawings. For ease of understanding, some technical features will be described from the perspectives of methods, apparatus, equipment, or systems in different embodiments. References Figure 1 The method includes:

[0100] Step 101: Obtain a first image containing the operator from the raw video stream captured by the camera, identify the operator's human body reference length, and measure the pixel length of the human body reference length in the first image;

[0101] The human body reference length is a linear distance with fixed physical dimensions on the operator's body;

[0102] Step 102: Determine a second image containing the operator's hand based on the first image, and obtain the coordinates of the key points of the hand in the second image;

[0103] Step 103: Determine the gesture interaction parameters based on the pixel length;

[0104] The gesture interaction parameters are adjusted according to the pixel length: when the operator moves away from the camera and the pixel length decreases, the gesture interaction parameters are increased; when the operator moves closer to the camera and the pixel length increases, the gesture interaction parameters are decreased.

[0105] Step 104: Based on the coordinates of the hand key points in the second image and the gesture interaction parameters, determine the intent of the interaction operation to generate control instructions for the gesture interaction operation.

[0106] In step 101, the original video stream refers to a continuous sequence of images directly output by the camera sensor without any cropping or enhancement processing. The first image refers to an image frame extracted from the original video stream that contains the operator's entire body or upper body; its resolution can be the same as the original video stream or it can be an image that has undergone preliminary scaling. The human body reference length corresponds to the anatomical structures of the operator's body with relatively stable physical dimensions, such as shoulder width, arm span, or height. Pixel length refers to the pixel distance between the two endpoints of this human body reference length in the image coordinate system. Recognition refers to the process of detecting and locating key points of the human body using computer vision algorithms. Measurement refers to the mathematical operation of calculating the Euclidean distance between two points.

[0107] In step 101, a dynamic calibration mechanism based on the human body's own dimensions is established, reducing the impact of distance changes on the interaction ratio. By acquiring the first image containing the operator, the baseline length of the human body (such as shoulder width) can be fully captured, providing reliable global context information for subsequent calculations. Using physically fixed human body parts as a benchmark, the distance to the user can be perceived without additional depth sensors.

[0108] In step 102, the second image refers to an image that is further located and extracted from the first image, mainly containing the operator's hand region. It can be a cropped view of the first image or a magnified view of that cropped view, or a region of interest image generated based on the information from the first image. Hand key points refer to the main feature points in the hand's skeletal structure. Coordinates refer to the two-dimensional positional information of these key points in the coordinate system of the second image.

[0109] Step 102 achieves refined processing from global to local, improving the resolution accuracy of hand features. The second image is determined based on the first image, utilizing global information to guide the local search and reducing false detections caused by blindly detecting hands in complex backgrounds. By obtaining coordinates from the second image, computational resources can be concentrated on a small area of ​​the hand, allowing the use of higher-resolution detection models, thus enabling clear identification of subtle finger movements even at long distances.

[0110] Specifically, the computer uses the human skeleton information detected in the first image to infer the approximate area of ​​the hand, then crops and enlarges that area to form a second image. Subsequently, the second image is input into a dedicated hand keypoint detection network, which outputs the coordinates of the keypoints.

[0111] Furthermore, in some embodiments of this application, the process of determining the second image also includes time-dimension smoothing, that is, combining the position information of the hand in the previous frame, predicting the possible area where the hand may appear in the first image of the current frame, and only cropping and generating the second image from the predicted area and its neighborhood, thereby reducing the amount of data processed by the image and improving the computer frame rate.

[0112] In step 103, gesture interaction parameters refer to the control variables used to convert hand gestures into screen operations. A smaller pixel length means the user is farther away, and the object in the image appears smaller; a larger pixel length means the user is closer, and the object in the image appears larger. Increasing or decreasing refers to adjusting the value according to a preset functional relationship (such as an inverse proportional relationship or a piecewise linear relationship).

[0113] This achieves distance-invariant interactive feel, ensuring consistent screen feedback regardless of the user's position or the range of their body movements. Furthermore, by monitoring real-time changes in pixel length, the computer can automatically detect the user's forward and backward movements without requiring manual calibration.

[0114] That is, when the user moves away, the parameter is increased to compensate for the small amount of hand pixel displacement caused by the distance; when the user moves closer, the parameter is decreased to prevent the small hand tremors caused by the close distance from being over-amplified, thereby improving the pain point of "dull at a distance and sensitive at a close distance" in the traditional fixed gain scheme.

[0115] Specifically, a standard pixel length value (corresponding to the optimal interaction distance) can be preset. When the current measured value is less than the standard value, a gain coefficient greater than 1 is calculated as the mapping ratio; otherwise, a gain coefficient less than 1 is calculated.

[0116] Furthermore, in some embodiments of this application, the adjustment of gesture interaction parameters is not completed instantaneously, but a first-order inertial element is introduced for a smooth transition. That is, the current interaction parameter value is a weighted average of the historical parameter value and the current calculated value. This can reduce the dizziness or discomfort caused by the sudden change in cursor speed when the user moves quickly back and forth.

[0117] In step 104, the intent of the interaction refers to the specific action logic that the user wants to perform, such as clicking to confirm, swiping to turn pages, dragging objects, or rotating views. Control commands refer to standardized signals sent to the display device or application for execution, such as mouse events, touch events, or specific Application Programming Interface (API) calls.

[0118] This completes the closed loop from visual perception to device control, enabling natural and seamless long-distance human-machine interaction. Applying dynamically calibrated parameters to coordinate analysis ensures the accuracy of the final generated control commands in physical space.

[0119] Specifically, by using hand coordinates and dynamic parameters in combination, the computer can recognize both static gestures (such as clenching a fist) and dynamic trajectories (such as waving a hand), and both recognition results are distance-compensated to ensure consistency of operation.

[0120] For example, the displacement vector of the hand key points relative to the previous frame is calculated and multiplied by the mapping ratio obtained in step 103 to obtain the cursor displacement on the screen. For example, the bending angle of the finger joints is detected; if it exceeds a threshold and remains stable, it is determined as a click intention, and a click command is generated.

[0121] The gesture interaction method of this application dynamically adjusts gesture interaction parameters by measuring the pixel length of the human body reference length in the image. Since the physical dimensions of the human body reference length are fixed, its pixel length can reflect changes in the imaging ratio in real time. Adjusting the parameters accordingly can automatically compensate for fluctuations in the mapping ratio caused by changes in distance, so that hand movements of the same physical amplitude produce consistent screen operation effects at different distances.

[0122] Specifically, dynamically adjusting gesture interaction parameters includes: acquiring a first image containing the operator from the original video stream and measuring the pixel length of the human body's baseline length (such as shoulder width), using this pixel length to dynamically determine the gesture interaction parameters, and then combining the hand key point coordinates to determine the interaction intent and generate control commands.

[0123] Without limitation, since the physical dimensions of the human body's baseline length are fixed, its pixel length in the image can reflect the actual distance between the operator and the camera and the imaging ratio in real time. Therefore, by dynamically adjusting the gesture interaction parameters (such as the mapping ratio or the trigger threshold) based on this pixel length, the "near is larger and far is smaller" perspective effect caused by changes in distance can be automatically compensated, so that no matter whether the user is standing far away or close, the same physical amplitude of hand movements can correspond to the same screen operation effect.

[0124] In other embodiments of this application, the human body reference length is the operator's shoulder width;

[0125] Measuring the pixel length of the human body reference length in the first image includes: identifying the coordinates of the operator's left and right shoulder peaks in the first image, and calculating the pixel distance between the two points as the pixel length.

[0126] Without restriction, the left and right shoulders are prominent and relatively stable lateral features of the upper body. The left and right acromion points are well-defined anatomical landmarks, easily located with high precision in pose estimation. Calculating the pixel distance between the two points as the pixel length is computationally simple and highly resistant to interference. Even with loose clothing, the position of the acromion point can be inferred relatively accurately using the algorithm. This choice fully utilizes the symmetry of ergonomics, making the baseline measurement less susceptible to the influence of arm swing.

[0127] Specifically, in some embodiments of this application, in order to improve the robustness of the measurement, the human body reference length can be not only the length of the line connecting the left and right shoulders, but also the diagonal length from the left acromion to the right hip joint, or a combination of head width and shoulder width. A comprehensive pixel length value is obtained by multi-feature fusion calculation to deal with the extreme case where one shoulder is occluded.

[0128] In some other embodiments of this application, the gesture interaction parameter is a gesture mapping ratio;

[0129] The step of determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, in order to generate control instructions for the gesture interaction operation, includes:

[0130] The coordinates of the key hand points in the second image are converted into operation positions on the screen using the gesture mapping ratio.

[0131] The intention of the interactive operation is determined based on the operation position on the screen, so as to generate control instructions for the gesture interaction operation.

[0132] In a broader sense, gesture mapping ratios act as a bridge between virtual screen space and real physical space. By multiplying image coordinates by this ratio, the physical movement of the hand is mapped to the movement of screen pixels. This approach is suitable for cursor-controlled interactions. Its advantage lies in establishing a linear correspondence between physical distance and screen distance, making the user feel as if they are directly manipulating objects on the screen.

[0133] In some other embodiments of this application, the gesture interaction parameter is a gesture trigger threshold;

[0134] The step of determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, in order to generate control instructions for the gesture interaction operation, includes:

[0135] Calculate the hand posture features or hand movement distance based on the coordinates of the key hand points in the second image;

[0136] The hand posture features or hand movement distance are compared with the gesture trigger threshold. When the gesture trigger threshold is exceeded, it is identified as a valid gesture action, and a corresponding control command for gesture interaction is generated.

[0137] Without limitation, a gesture trigger threshold is used to determine whether an action is valid and to prevent accidental touches. At long distances, the calculation of hand features may have slight fluctuations due to image noise and compression artifacts. By dynamically adjusting the threshold (e.g., lowering the angle determination threshold at long distances, because angle calculation errors are large at distances; or increasing the displacement determination threshold at long distances to prevent minor tremors from being misjudged as sliding), the accuracy of recognition can be improved.

[0138] In other embodiments of this application, determining the gesture interaction parameters based on the pixel length includes:

[0139] Establish the correspondence between the physical area corresponding to the operator's body reference length and the screen display area;

[0140] This allows the hand to move within the human body's baseline length range, corresponding to the operation displacement of the entire screen area or a portion of the screen area.

[0141] In essence, this is a normalized mapping strategy. It maps the natural range of motion of the human body (such as shoulder width) to the full resolution or a portion of the screen. That is, users only need to wave their hands within shoulder width to cover the entire screen. Regardless of how far the user stands, this "shoulder width" baseline remains constant, thus the feel of the operation is relatively consistent. This method reduces the learning curve for users and allows for the formation of muscle memory.

[0142] In other embodiments of this application, obtaining the coordinates of key hand points in the second image includes:

[0143] Identify the operator's hand orientation and determine whether the operator's hand is left-handed or right-handed;

[0144] Calculate the coordinates of the hand root in the second image, and establish a local coordinate system for the hand using the hand root as an anchor point;

[0145] Based on the aforementioned local hand coordinate system, the coordinates of other key points of the hand, excluding the base of the hand, are extracted.

[0146] In a non-restricted sense, the base of the hand, i.e., the wrist, can be used as an anchor point to establish a local coordinate system. This allows for the conversion of absolute coordinates to relative coordinates, reducing the impact of overall body movement. Identifying the left and right hands helps the system distinguish between two-handed operations or filter out interference from the non-dominant hand. This local coordinate system method improves the purity of feature extraction in complex backgrounds or multi-person scenes, eliminating interference from other body parts.

[0147] In some other embodiments of this application, obtaining the coordinates of key hand points in the second image includes a jump compensation step:

[0148] Get the original coordinates of the hand in the current frame;

[0149] The predicted hand coordinates of the current frame are calculated using a prediction algorithm based on the motion trends of the previous N frames; where N is an integer greater than or equal to 2.

[0150] The original hand coordinates are compared with the predicted hand coordinates. If the deviation between the two exceeds a preset human motion threshold, the current frame is determined to be abnormal, and the predicted hand coordinates are used as the coordinates of the hand key points in the second image. If the deviation is within the normal range, the original hand coordinates are used, and the internal parameters of the prediction algorithm are updated using the original hand coordinates.

[0151] In a broader sense, jump compensation is a technique to reduce recognition jitter. Due to changes in lighting or occlusion, the recognition model may occasionally output incorrect coordinates (jumps). The prediction algorithm, based on the principle of motion continuity, calculates reasonable coordinates. By comparing the original and predicted values, the system can automatically identify and remove abnormal frames, filling them with the predicted values ​​to ensure smooth trajectory. Simultaneously, normal frames are used to update algorithm parameters, making predictions more accurate.

[0152] In other embodiments of this application, after determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters to generate control instructions for the gesture interaction operation, a smoothing process step is further included:

[0153] The adaptive filter cutoff frequency is dynamically adjusted based on the movement speed of the key hand points: when the movement speed is greater than a preset speed threshold, the filter cutoff frequency is increased to reduce signal delay; when the movement speed is less than or equal to the preset speed threshold, the filter cutoff frequency is decreased to suppress signal jitter.

[0154] Unrestricted, adaptive filters mitigate the trade-off between smoothness and latency. Traditional fixed filters are either too laggy (high latency) or too jittery (high noise). This solution dynamically adjusts based on speed: less filtering for fast movements to ensure responsiveness; more filtering for slow movements to ensure stability. This allows the cursor on the terminal screen to move accordingly when the user swings their arm quickly, while maintaining cursor stability during fine-tuning.

[0155] In some other embodiments of this application, when generating the control commands for gesture interaction, a step to prevent accidental touches is also included:

[0156] Monitor changes in the state of hand gestures, and enter a cooldown period after a valid trigger command is detected;

[0157] During the cooling period, subsequent identical trigger signals are ignored to prevent repeated execution of commands due to hand tremors.

[0158] Unrestricted, the accidental touch prevention step prevents continuous triggering caused by physiological hand tremors or recognition fluctuations by introducing time-dimensional mutual exclusion logic. The cool-down period provides a brief protection window, enhancing the determinism of operation.

[0159] In some other embodiments of this application, before determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, an orientation calibration step is further included:

[0160] Identify the operator's body orientation angle;

[0161] Based on the body's orientation angle, adjust the starting point or direction of the operation area on the screen display area so that when the operator stands sideways, the direction of hand movement is consistent with the operation direction on the screen display area.

[0162] The determination of the intent of the interactive operation specifically includes: determining the intent of the interactive operation based on the adjusted starting point or direction of the operation area.

[0163] In real-world scenarios, users rarely stand directly facing the screen. The orientation calibration step dynamically rotates the mapped coordinate system by detecting the angle between the body's principal axis and the screen normal. This way, even if a user stands at a 45-degree angle and waves their hand forward, the cursor will still move forward on the screen, rather than diagonally.

[0164] By combining global human body images with local hand images for collaborative processing, the system can adapt to changes in the operator's position and posture, thereby improving the stability and consistency of long-distance gesture interaction.

[0165] In addition, this method extracts baseline features based on a global image containing the operator and combines it with local hand images for collaborative processing. It can effectively adapt to operation scenarios with different body orientations, reduce orientation misjudgment caused by posture deviation, and thus improve the stability, consistency and naturalness of long-distance gesture interaction without requiring the user to deliberately maintain a specific position or posture.

[0166] Specifically, when the operator stands sideways, the starting point of the operating area may be off to one side of the screen (e.g., the end of the operating direction), and the operating direction may also be diagonal. Adjustments can be made by moving the starting point of the operating area to the beginning of the operating direction, or by adjusting the operating direction to be horizontal or vertical.

[0167] Adjusting the starting point or direction of the operation area on the screen display area based on the body's orientation angle can be done by adjusting the starting point or direction of the operation area on the screen display area when the body's orientation angle exceeds a preset range.

[0168] In some other embodiments of this application, before determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, a gaze calibration step is further included:

[0169] Identify the operator's eye movement trajectory;

[0170] By combining the human eye gaze trajectory with the human body reference length, a two-layer calibration mechanism is constructed, wherein: the first layer determines the overall mapping ratio based on the human body reference length, and the second layer determines the operation area based on the human eye gaze trajectory;

[0171] The determination of the intent of the interactive operation specifically includes: determining the intent of the interactive operation based on the overall mapping ratio and operation area determined by the dual-layer calibration mechanism.

[0172] Without limitations, gaze calibration introduces an attention mechanism. The first layer, shoulder width calibration, addresses distance issues, while the second layer, gaze calibration, addresses focus issues. The system dynamically moves highly sensitive operation areas to the user's gaze area. For example, when it detects that the user's gaze is focused on a certain area of ​​the screen and their hand makes a grasping gesture, it prioritizes interpreting the intent as a drag operation on an object in that area, rather than a global cursor movement, thus achieving multimodal intent understanding. This two-layer mechanism simulates the human instinct for hand-eye coordination, improving the efficiency of fine-grained operations on large screens.

[0173] In other embodiments of this application, determining a second image containing the operator's hand based on the first image includes an image enhancement step:

[0174] Extract a third image containing the hand from the first image;

[0175] The third image is subjected to detail magnification and enhancement processing to obtain an enhanced image with higher clarity than the original video stream, and the enhanced image is used as the second image;

[0176] Based on the second image, key points of the hand are identified.

[0177] Without limitations, hands have very few pixels at long distances, making direct recognition difficult. The image enhancement step first extracts a low-resolution local image (the third image), then reconstructs details using a super-resolution algorithm to generate a high-resolution second image. This is equivalent to a telephoto lens at the software level. This step improves the visibility of finger joints at long distances, contributing to high-precision recognition.

[0178] In other embodiments of this application, the detail magnification and enhancement processing includes:

[0179] The pixels of the third image are reconstructed using a weighted interpolation algorithm;

[0180] A window function is used to limit the range of neighboring pixels involved in the calculation, and smoothing weights are assigned to pixels at different locations to suppress artifacts at image edges.

[0181] Unrestricted, weighted interpolation algorithms generate new pixels by considering the contribution of surrounding pixels, resulting in a sharper image than ordinary stretching. The window function limits the computational range and smooths the transition, preventing jagged edges or halos (artifacts) at the finger edges. This approach improves resolution while maintaining sharp and natural edges, avoiding computational interference from false textures in the recognition algorithm.

[0182] In some other embodiments of this application, before acquiring the first image containing the operator from the raw video stream captured from the camera, an operator locking step is further included:

[0183] Track multiple target objects in the original video stream and assign them unique identifiers;

[0184] Calculate the overlap between each target object and the preset interaction area, the dwell time, and the human body orientation score to obtain the interaction intent score;

[0185] Select the target object with the highest interaction intent score as the operator.

[0186] In multi-user scenarios, however, it is essential to identify only one operator. This step calculates an interaction intent score by comprehensively evaluating the user's posture and actions. The user with the highest interaction intent score is identified as the operator or master user, while the actions of others are ignored. This effectively improves the interaction interference problem in multi-user environments, ensuring that the system only responds to the user with the strongest interaction intention.

[0187] In some other embodiments of this application, obtaining the coordinates of key hand points in the second image further includes a hand orientation correction step:

[0188] Obtain the angle between the vertical axis of the palm and the vertical axis of the second image; wherein, the vertical axis of the palm is the direction extending from the wrist to the fingers;

[0189] The image region containing the hand is rotated and corrected according to the included angle, so that the longitudinal direction of the palm in the corrected hand image is consistent with the preset standard direction, so as to accurately extract the key points of the hand.

[0190] Unrestricted, hand recognition models typically perform best when the palm is vertical. This step detects the hand's tilt angle and rotates the image to straighten it before inputting it into the model. This is a standardization step at the data preprocessing level. By standardizing the input pose, the requirements for the recognition model's generalization ability are reduced, improving the recognition success rate under various non-standard gestures. The second image's vertical axis refers to the vertical or height direction of the second image, i.e., the direction from the top edge to the bottom edge of the image (Y-axis).

[0191] In other embodiments of this application, the key points of the hand include the wrist, palm, and the joints of each finger;

[0192] When determining the intent of the interactive operation to generate control instructions for the gesture interactive operation, at least one of the following hand state features is also utilized: the degree of finger flexion, used to characterize the opening and closing state of the fingers.

[0193] The direction the palm faces is used to indicate the spatial orientation of the hand;

[0194] The number of fingers extended is used to represent the number of fingers that can be straightened.

[0195] And the relative instantaneous velocity of each finger joint, used to characterize the speed of hand movement.

[0196] Unrestricted, a rich set of hand gesture features supports a variety of interactive commands. The degree of finger bending can be used to confirm a fist; the direction of the palm can be used to switch modes by flipping; the number of fingers can be used for quick menus; and the speed can be used to distinguish the strength of the intent.

[0197] In some other embodiments of this application, the prediction algorithm is a Kalman filter algorithm;

[0198] The step of using a prediction algorithm to calculate the predicted hand coordinates for the current frame based on the motion trends of the previous N frames includes:

[0199] Based on the average velocity and acceleration of hand movement in historical frames, the global displacement of the hand is calculated in time series to obtain the predicted coordinates of the hand in the current frame.

[0200] In an unrestricted manner, the Kalman filter is an optimal estimation algorithm that uses velocity and acceleration information to predict the position at the next moment. In jump compensation, it provides a theoretical "truth" reference.

[0201] In some other embodiments of this application, the adaptive filter is a frequency conversion low-pass filter.

[0202] In an unrestricted manner, the variable frequency low-pass filter achieves self-adaptation by dynamically changing the cutoff frequency. High-frequency signals (rapid movement) pass through, while low-frequency signals (minor jitter) are filtered out. This frequency domain processing method precisely separates the effective operating signal from the noise signal, achieving a balance between delay and stability at the physical level.

[0203] In some other embodiments of this application, the cooling time is 300 milliseconds to 800 milliseconds.

[0204] Without limitation, the cooling time range is based on empirical values ​​set according to human reaction time and common jitter frequencies. Too short a time will not filter jitter, while too long a time will cause sluggish operation. This specific range has been verified through extensive experiments and can provide a good anti-mistouch experience in most scenarios.

[0205] In some other embodiments of this application, the step of identifying the operator's body reference length further includes extracting the operator's neck coordinates and torso center coordinates:

[0206] Based on the neck coordinates, torso center coordinates, and the coordinates of the two endpoints constituting the human body reference length, a human upper body skeleton model is constructed.

[0207] The geometric constraints of the human upper body skeleton model are used to verify and correct the measurement results of the human body reference length.

[0208] However, under extreme occlusion or lighting conditions, relying solely on shoulder detection may lead to errors. Therefore, by introducing the center points of the neck and torso, a triangular or polygonal skeletal structure is formed. Using geometric constraints of human anatomy (such as a relatively fixed shoulder-neck distance and torso symmetry), the reliability of shoulder width measurement can be verified. If the measured value deviates too much from the geometric constraints, the system can automatically perform interpolation correction using the neck and torso points.

[0209] This further improves the robustness of human body reference length measurement, prevents parameter drift of the entire interactive system due to key point detection failure, and enhances the system's fault tolerance in complex environments. In some other embodiments of this application, after generating the control command for gesture interaction, an instruction mapping adaptation step is also included: identifying the type of the currently running application; mapping the control command to a specific input protocol supported by the application according to the application type; wherein, if the application is demo software, the swipe gesture is mapped to a page-turning command; if the application is map software, the swipe gesture is mapped to a view panning command.

[0210] In general, generic mouse simulation commands don't perform well in some specialized software. Therefore, it's designed to perform semantic-level command translation for the same gesture across different applications. For example, in PowerPoint, a horizontal wave triggers the "Next Page" API instead of simulating a mouse click on the bottom right button.

[0211] This enables deep application-level adaptation, making gesture interaction not just a replacement for the mouse, but an intelligent input method with semantic understanding capabilities, improving operational efficiency and user experience in specific scenarios.

[0212] To better understand the gesture interaction method of this application's embodiments, a more specific embodiment is described below. In the more specific embodiment, refer to... Figure 2 The gesture interaction method includes three stages: preprocessing, calculation, and postprocessing.

[0213] Prior to the preprocessing stage, the following are also included:

[0214] Step 201: Obtain the raw video stream. This involves acquiring the raw video stream captured by the camera, which serves as the raw data for preprocessing.

[0215] The preprocessing stage includes:

[0216] Step 202: Human Target Detection. Each frame of the original video stream is analyzed using a deep learning model to identify and locate the bounding boxes of all human figures in the frame. This step is a prerequisite for constructing the "first image," ensuring that the system can focus on the area where the operator is located, corresponding to the initial screening step of "obtaining the first image containing the operator" mentioned earlier.

[0217] Step 203: Multi-target human tracking. A unique identifier is assigned to each detected human target, and their motion trajectory is tracked across frames. This step supports the "operator locking" mechanism, providing a temporal data foundation for subsequent calculation of the "interaction intent score" by continuously tracking the position, velocity, and attitude of each target.

[0218] Step 204: Image Enlargement. The local area containing the hand is enhanced and enlarged to generate a clearer image. This operation, referred to as the "image enhancement step" above, aims to improve the recognizability of hand features at a distance, creating conditions for subsequent high-precision keypoint extraction. The output is the "second image" used for hand analysis.

[0219] The calculation phase includes:

[0220] Step 205: Hand Tracking. Based on global human skeleton information, the hand position is located and tracked in consecutive frames. This step utilizes the human pose estimation results from the previous stage to narrow down the hand search range, improve efficiency and robustness, and is a dynamic implementation of "determining the second image containing the hand".

[0221] Step 206: Calculate the coordinates of the hand root. The hand root, i.e., the center point of the wrist, is the pivot of hand movement. Calculating its coordinates is used to establish a local coordinate system for the hand, thereby representing other key points as relative positions and effectively eliminating interference from overall body movement. This corresponds to the technical details of "establishing a local coordinate system for the hand using the hand root as an anchor point" mentioned earlier.

[0222] Step 207: Correct the hand image. Based on the angle between the palm's longitudinal axis and the image coordinates, rotate the hand region to align the palm's principal axis with the standard direction. This step, also known as the "hand orientation correction step" mentioned earlier, improves the accuracy of the keypoint detection model by preprocessing and standardizing the input pose.

[0223] Step 208: Hand Keypoint Calculation. A keypoint detection network is run on the corrected hand image, outputting the coordinates of multiple feature points, including the palm and knuckles. This is the core calculation step for obtaining the coordinates of "hand keypoints in the second image," supporting subsequent interactive intent recognition.

[0224] Step 209: Kalman Filter Predicts Future Coordinates. Based on the hand motion state (position, velocity, acceleration) of historical frames, the theoretical coordinates of the current frame are calculated using the Kalman filter algorithm. This step constitutes the prediction module for "jump compensation," providing a benchmark reference for determining whether the recognition result is abnormal.

[0225] Step 210: Calculate the coordinates of key hand points. This refers to the original hand coordinates output by the model in the current frame, which is consistent with step 208, but in this process it specifically refers to the original observation values ​​to be verified, used to compare with the predicted values ​​in step 209.

[0226] Step 211: Determine if there is a frame jump. If yes, proceed to step 212; otherwise, proceed to step 213. Compare the deviation between the original coordinates and the predicted hand coordinates to see if it exceeds the physiological limit threshold of human movement. If it exceeds the limit, it is determined to be an abnormal recognition (frame jump). This judgment logic is the key decision of the "frame jump compensation step," ensuring that the system can automatically filter out erroneous outputs caused by occlusion or sudden changes in lighting.

[0227] Step 212: Use predicted coordinates. When a frame jump is detected, the unreliable original coordinates are discarded, and the Kalman filter prediction values ​​are used as the final hand keypoint coordinates. This ensures the continuity and smoothness of the trajectory.

[0228] Step 213: Use the original coordinates. If the deviation is within a reasonable range, the original coordinates are considered reliable. These values ​​are then adopted as the final coordinates, and the state parameters of the Kalman filter are updated accordingly, resulting in more accurate subsequent predictions. This demonstrates the system's adaptive learning capability under normal conditions.

[0229] The post-processing stage includes:

[0230] Step 214: Calculate hand metrics. Based on the final hand keypoint coordinates, calculate state features such as finger flexion degree, palm orientation, number of extensions, and joint speed. These metrics are used to characterize the user's specific gesture semantics, supporting the multi-dimensional recognition mechanism described above of "determining interaction intent using hand state features."

[0231] Step 215: Calculate the mapping area and human orientation. Simultaneously analyze the whole-body posture, extract the body orientation angle, and combine it with the gaze trajectory (if any) to determine the currently valid screen operation area. This step integrates "orientation calibration" and "gaze calibration," ensuring that the operation direction remains consistent with the user's experience even when the user is turned to the side or their gaze shifts.

[0232] Step 216: Sensitivity and Status Parameter Adjustment. Based on the pixel length corresponding to the human body's baseline length (e.g., shoulder width), dynamically adjust the gesture mapping ratio or trigger threshold; simultaneously switch between high-precision and low-latency modes according to hand movement speed. This is the specific execution of the previously mentioned "determining gesture interaction parameters based on pixel length" and "adaptive sensitivity."

[0233] Step 217: Determine the gesture and execute a cooldown mutual exclusion operation. Based on hand indicators and current state parameters, identify gesture intentions such as clicks and swipes; once a valid command is triggered, immediately start a cooldown timer to block similar signals for a set time. This process fully embodies the mutual exclusion logic of the "anti-accidental touch step," preventing repeated execution due to jitter.

[0234] Step 218: Coordinate smoothing using a One Euro filter. The One Euro filter is an adaptive low-pass filter designed for real-time signal smoothing, balancing jitter and delay by dynamically adjusting the cutoff frequency. Its core logic is: lowering the cutoff frequency at low speeds to suppress high-frequency noise (reducing jitter), and raising the cutoff frequency at high speeds to preserve signal variations (reducing delay). As can be seen, the One Euro filter achieves a dynamic balance between jitter and delay through a simple adaptive mechanism, combining low computational overhead and high practicality.

[0235] The post-processing stage includes:

[0236] Step 219: Execute the command. Send the fully post-processed and optimized control commands to the display terminal or application to drive the screen to complete operations such as page turning, clicking, and dragging, thus completing the closed loop from visual perception to device control.

[0237] Example 2

[0238] This application provides a gesture interaction device 30, see reference. Figure 3 The gesture interaction device 30 includes:

[0239] The measurement module 31 is used to acquire a first image containing the operator from the original video stream captured by the camera, identify the human body reference length of the operator, and measure the pixel length of the human body reference length in the first image; wherein, the human body reference length is a linear distance with fixed physical dimensions on the operator's body;

[0240] The acquisition module 32 is used to determine a second image containing the operator's hand based on the first image, and to acquire the coordinates of key points of the hand in the second image;

[0241] The first determining module 33 is used to determine gesture interaction parameters based on the pixel length; wherein, the gesture interaction parameters are adjusted as the pixel length changes: when the operator moves away from the camera and the pixel length decreases, the gesture interaction parameters are increased; when the operator moves closer to the camera and the pixel length increases, the gesture interaction parameters are decreased.

[0242] The second determining module 34 is used to determine the intention of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, so as to generate control instructions for the gesture interaction operation.

[0243] In other embodiments of this application, the human body reference length is the operator's shoulder width;

[0244] The measurement module 31 is also used to: identify the coordinates of the operator's left shoulder peak and right shoulder peak in the first image, and calculate the pixel distance between the two points as the pixel length.

[0245] Without restriction, the left and right shoulders are prominent and relatively stable lateral features of the upper body. The left and right acromion points are well-defined anatomical landmarks, easily located with high precision in pose estimation. Calculating the pixel distance between the two points as the pixel length is computationally simple and highly resistant to interference. Even with loose clothing, the position of the acromion point can be inferred relatively accurately using the algorithm. This choice fully utilizes the symmetry of ergonomics, making the baseline measurement less susceptible to the influence of arm swing.

[0246] Specifically, in some embodiments of this application, in order to improve the robustness of the measurement, the human body reference length can be not only the length of the line connecting the left and right shoulders, but also the diagonal length from the left acromion to the right hip joint, or a combination of head width and shoulder width. A comprehensive pixel length value is obtained by multi-feature fusion calculation to deal with the extreme case where one shoulder is occluded.

[0247] In some other embodiments of this application, the gesture interaction parameter is a gesture mapping ratio;

[0248] The second determining module 34 is also used for:

[0249] The coordinates of the key hand points in the second image are converted into operation positions on the screen using the gesture mapping ratio.

[0250] The intention of the interactive operation is determined based on the operation position on the screen, so as to generate control instructions for the gesture interaction operation.

[0251] In a broader sense, gesture mapping ratios act as a bridge between virtual screen space and real physical space. By multiplying image coordinates by this ratio, the physical movement of the hand is mapped to the movement of screen pixels. This approach is suitable for cursor-controlled interactions. Its core advantage lies in establishing a linear correspondence between physical distance and screen distance, making the user feel as if they are directly manipulating objects on the screen.

[0252] In some other embodiments of this application, the gesture interaction parameter is a gesture trigger threshold;

[0253] The second determining module 34 is also used for:

[0254] Calculate the hand posture features or hand movement distance based on the coordinates of the key hand points in the second image;

[0255] The hand posture features or hand movement distance are compared with the gesture trigger threshold. When the gesture trigger threshold is exceeded, it is identified as a valid gesture action, and a corresponding control command for gesture interaction is generated.

[0256] Without limitation, a gesture trigger threshold is used to determine the validity of an action and prevent accidental touches. At long distances, the calculation of hand features may exhibit slight fluctuations due to image noise and compression artifacts. By dynamically adjusting the threshold (e.g., lowering the angle determination threshold at long distances, as angle calculation errors are large at distances; or increasing the displacement determination threshold at long distances to prevent minor tremors from being misinterpreted as sliding), the accuracy of recognition can be improved. This dynamic thresholding mechanism effectively balances sensitivity and noise resistance, reducing the phenomenon of near-immobility or random movement.

[0257] In some other embodiments of this application, the first determining module 33 is further configured to:

[0258] Establish the correspondence between the physical area corresponding to the operator's body reference length and the screen display area;

[0259] This allows the hand to move within the human body's baseline length range, corresponding to the operation displacement of the entire screen area or a portion of the screen area.

[0260] In essence, this is a normalized mapping strategy. It maps the natural range of motion of the human body (such as shoulder width) to the full resolution or a portion of the screen. That is, users only need to wave their hands within a shoulder-width range to cover the entire screen. Regardless of how far the user stands, this physical sensation of "shoulder width" remains constant, thus ensuring a relatively stable tactile feel. This method reduces the learning curve for users, fostering muscle memory.

[0261] In other embodiments of this application, the acquisition module 32 is further configured to:

[0262] Identify the operator's hand orientation and determine whether the operator's hand is left-handed or right-handed;

[0263] Calculate the coordinates of the hand root in the second image, and establish a local coordinate system for the hand using the hand root as an anchor point;

[0264] Based on the aforementioned local hand coordinate system, the coordinates of other key points of the hand, excluding the base of the hand, are extracted.

[0265] In a broader sense, the wrist, the base of the hand, is the pivot point for hand movements. Establishing a local coordinate system using it as an anchor point allows for the conversion of absolute coordinates to relative coordinates, reducing the impact of overall body movement. Identifying the left and right hands helps the system distinguish between two-handed operations or mask interference from the non-dominant hand. This local coordinate system method improves the purity of feature extraction in complex backgrounds or multi-person scenes.

[0266] In other embodiments of this application, the acquisition module 32 is further configured to:

[0267] Get the original coordinates of the hand in the current frame;

[0268] The predicted hand coordinates of the current frame are calculated using a prediction algorithm based on the motion trends of the previous N frames; where N is an integer greater than or equal to 2.

[0269] The original hand coordinates are compared with the predicted hand coordinates. If the deviation between the two exceeds a preset human motion threshold, the current frame is determined to be abnormal, and the predicted hand coordinates are used as the coordinates of the hand key points in the second image. If the deviation is within the normal range, the original hand coordinates are used, and the internal parameters of the prediction algorithm are updated using the original hand coordinates.

[0270] In general, jump compensation is a key technology for improving jitter detection. Due to changes in lighting or occlusion, the detection model may occasionally output incorrect coordinates (jumps). The prediction algorithm, based on the principle of motion continuity, calculates reasonable coordinates. By comparing the original and predicted values, the system can automatically identify and remove abnormal frames, filling them with the predicted values ​​to ensure smooth trajectory. Simultaneously, normal frames are used to update algorithm parameters, making predictions more accurate.

[0271] In other embodiments of this application, the apparatus further includes a first processing module, the first processing module being configured to:

[0272] The adaptive filter cutoff frequency is dynamically adjusted based on the movement speed of the key hand points: when the movement speed is greater than a preset speed threshold, the filter cutoff frequency is increased to reduce signal delay; when the movement speed is less than or equal to the preset speed threshold, the filter cutoff frequency is decreased to suppress signal jitter.

[0273] Unrestricted, adaptive filters mitigate the trade-off between smoothness and latency. Traditional fixed filters are either too laggy (high latency) or too jittery (high noise). This solution dynamically adjusts based on speed: less filtering for fast movements to ensure responsiveness; more filtering for slow movements to ensure stability. This allows the cursor to follow rapidly when the user swings their arm, while remaining stable during fine-tuning.

[0274] In some other embodiments of this application, the second determining module 34 is further configured to:

[0275] Monitor changes in the state of hand gestures, and enter a cooldown period after a valid trigger command is detected;

[0276] During the cooling period, subsequent identical trigger signals are ignored to prevent repeated execution of commands due to hand tremors.

[0277] Unrestricted, the accidental touch prevention step prevents continuous triggering due to physiological hand tremors or recognition fluctuations by introducing time-dimensional mutual exclusion logic. The cooldown period serves as a brief protective window. This is particularly important for click-based operations, ensuring that a single gesture triggers only one command, thus improving the determinism of the operation.

[0278] In other embodiments of this application, the apparatus further includes a second processing module, the second processing module being used for:

[0279] Identify the operator's body orientation angle;

[0280] Based on the body's orientation angle, adjust the starting point or direction of the operation area on the screen display area so that when the operator stands sideways, the direction of hand movement is consistent with the operation direction on the screen display area.

[0281] The second determining module 34 is also used to: determine the intent of the interactive operation based on the adjusted starting point or direction of the operation area.

[0282] In real-world scenarios, users rarely stand directly facing the screen. The orientation calibration step dynamically rotates the mapped coordinate system by detecting the angle between the body's principal axis and the screen normal. This way, even if a user stands at a 45-degree angle and waves their hand forward, the cursor will still move forward on the screen, rather than diagonally.

[0283] In addition, this method extracts baseline features based on a global image containing the operator and combines it with local hand images for collaborative processing. It can effectively adapt to operation scenarios with different body orientations, reduce orientation misjudgment caused by posture deviation, and thus improve the stability, consistency and naturalness of long-distance gesture interaction without requiring the user to deliberately maintain a specific position or posture.

[0284] In other embodiments of this application, the apparatus further includes a third processing module, the third processing module being used for:

[0285] Identify the operator's eye movement trajectory;

[0286] By combining the human eye gaze trajectory with the human body reference length, a two-layer calibration mechanism is constructed, wherein: the first layer determines the overall mapping ratio based on the human body reference length, and the second layer determines the operation area based on the human eye gaze trajectory;

[0287] The determination of the intent of the interactive operation specifically includes: determining the intent of the interactive operation based on the overall mapping ratio and operation area determined by the dual-layer calibration mechanism.

[0288] Without limitations, gaze calibration introduces an attention mechanism. The first layer, shoulder width calibration, addresses distance issues, while the second layer, gaze calibration, addresses focus issues. The system dynamically moves highly sensitive operation areas to the user's gaze area. For example, when it detects that the user's gaze is focused on a certain area of ​​the screen and their hand makes a grasping gesture, it prioritizes interpreting the intent as a drag operation on an object in that area, rather than a global cursor movement, thus achieving multimodal intent understanding. This two-layer mechanism simulates the human instinct for hand-eye coordination, improving the efficiency of fine-grained operations on large screens.

[0289] In other embodiments of this application, the acquisition module 32 is further configured to:

[0290] Extract a third image containing the hand from the first image;

[0291] The third image is subjected to detail magnification and enhancement processing to obtain an enhanced image with higher clarity than the original video stream, and the enhanced image is used as the second image;

[0292] Based on the second image, key points of the hand are identified.

[0293] Without limitations, hands have very few pixels at long distances, making direct recognition difficult. The image enhancement step first extracts a low-resolution local image (the third image), then reconstructs details using a super-resolution algorithm to generate a high-resolution second image. This step is equivalent to implementing local image magnification and detail reconstruction at the software level, improving the visibility of finger joints at long distances, which is a prerequisite for high-precision recognition.

[0294] In other embodiments of this application, the acquisition module 32 is further configured to:

[0295] The pixels of the third image are reconstructed using a weighted interpolation algorithm;

[0296] A window function is used to limit the range of neighboring pixels involved in the calculation, and smoothing weights are assigned to pixels at different locations to suppress artifacts at image edges.

[0297] Unrestricted, weighted interpolation algorithms generate new pixels by considering the contribution of surrounding pixels, resulting in a sharper image than ordinary stretching. The window function limits the computational range and smooths the transition, preventing jagged edges or halos (artifacts) at the finger edges. This approach improves resolution while maintaining sharp and natural edges, avoiding misleading the recognition algorithm with false textures.

[0298] In other embodiments of this application, the device further includes an operator locking module, the operator locking module being used for:

[0299] Track multiple target objects in the original video stream and assign them unique identifiers;

[0300] Calculate the overlap between each target object and the preset interaction area, the dwell time, and the human body orientation score to obtain the interaction intent score;

[0301] Select the target object with the highest interaction intent score as the operator.

[0302] In unrestricted scenarios with multiple users, it is essential to identify only one operator. This step calculates an interaction intent score by comprehensively evaluating who stands in the interaction area, who stands there the longest, and who faces the screen. The user with the highest interaction intent score is designated as the operator or master user, while the actions of others are ignored. This effectively improves the control conflict problem in multi-user environments, ensuring that the system only responds to the user with the strongest interaction intention.

[0303] In other embodiments of this application, the acquisition module 32 is further configured to:

[0304] Obtain the angle between the vertical axis of the palm and the vertical axis of the second image; wherein, the vertical axis of the palm is the direction extending from the wrist to the fingers;

[0305] The image region containing the hand is rotated and corrected according to the included angle, so that the longitudinal direction of the palm in the corrected hand image is consistent with the preset standard direction, so as to accurately extract the key points of the hand.

[0306] Unrestricted, hand recognition models typically perform best when the palm is perpendicular. This step detects the hand's tilt angle and rotates the image to straighten it before inputting it into the model. This falls under the category of standardization at the data preprocessing level. By standardizing the input pose, the requirements for the recognition model's generalization ability are reduced, improving the recognition success rate for various non-standard gestures.

[0307] In other embodiments of this application, the key points of the hand include the wrist, palm, and the joints of each finger;

[0308] The second determining module 34 is further configured to: when determining the intent of the interactive operation to generate control instructions for the gesture interactive operation, utilize at least one of the following hand state features:

[0309] The degree of finger bending is used to characterize the open and closed state of the fingers;

[0310] The direction the palm faces is used to indicate the spatial orientation of the hand;

[0311] The number of fingers extended is used to represent the number of fingers that can be straightened.

[0312] And the relative instantaneous velocity of each finger joint, used to characterize the speed of hand movement.

[0313] Unrestricted, rich hand state features support diverse interaction commands. Finger bending degree can be used for fist confirmation; palm orientation can be used for flipping to switch modes; number of fingers can be used for quick menus; speed can be used to distinguish the strength of intent. The fusion of multi-dimensional features makes the interaction methods richer and more refined.

[0314] In some other embodiments of this application, the prediction algorithm is a Kalman filter algorithm;

[0315] The acquisition module 32 is further configured to:

[0316] Based on the average velocity and acceleration of hand movement in historical frames, the global displacement of the hand is calculated in time series to obtain the predicted coordinates of the hand in the current frame.

[0317] In an unrestricted manner, the Kalman filter is an optimal estimation algorithm that uses velocity and acceleration information to predict the position at the next moment. In jump compensation, it provides a theoretical "truth" reference.

[0318] In some other embodiments of this application, the adaptive filter is a frequency conversion low-pass filter.

[0319] In an unrestricted manner, the variable frequency low-pass filter achieves self-adaptation by dynamically changing the cutoff frequency. High-frequency signals (rapid movement) pass through, while low-frequency signals (minor jitter) are filtered out. This frequency domain processing method precisely separates the effective operating signal from the noise signal, achieving a balance between delay and stability at the physical level.

[0320] In some other embodiments of this application, the cooling time is 300 milliseconds to 800 milliseconds.

[0321] Without limitation, the cooling time range is based on empirical values ​​set according to human reaction time and common jitter frequencies. Too short a time will not filter jitter, while too long a time will cause sluggish operation. This specific range has been verified through extensive experiments and can provide a good anti-mistouch experience in most scenarios.

[0322] In other embodiments of this application, the measurement module 31 is further configured to:

[0323] Based on the neck coordinates, torso center coordinates, and the coordinates of the two endpoints constituting the human body reference length, a human upper body skeleton model is constructed.

[0324] The geometric constraints of the human upper body skeleton model are used to verify and correct the measurement results of the human body reference length.

[0325] However, under extreme occlusion or lighting conditions, relying solely on shoulder detection may lead to errors. Therefore, by introducing the center points of the neck and torso, a triangular or polygonal skeletal structure is formed. Using geometric constraints of human anatomy (such as a relatively fixed shoulder-neck distance and torso symmetry), the reliability of shoulder width measurement can be verified. If the measured value deviates too much from the geometric constraints, the system can automatically perform interpolation correction using the neck and torso points.

[0326] This further improves the robustness of human body reference length measurement, prevents parameter drift of the entire interactive system due to key point detection failure, and enhances the system's fault tolerance in complex environments.

[0327] In some other embodiments of this application, the apparatus further includes a fourth processing module, the fourth processing module being configured to: identify the type of the currently running application; and map the control commands to a specific input protocol supported by the application according to the type of the application; wherein, if the application is demo software, the swipe gesture is mapped to a page-turning command; and if the application is map software, the swipe gesture is mapped to a view panning command.

[0328] In general, generic mouse simulation commands don't perform well in some specialized software. Therefore, it's designed to perform semantic-level command translation for the same gesture across different applications. For example, in PowerPoint, a horizontal wave directly triggers the "Next Page" API, instead of simulating a mouse click on the bottom right corner button.

[0329] This enables deep application-level adaptation, making gesture interaction not just a replacement for the mouse, but an intelligent input method with semantic understanding capabilities, improving operational efficiency and user experience in specific scenarios.

[0330] The modules included in this embodiment can be implemented using a processor in a computer; alternatively, they can be implemented using logic circuits in a computer. The processor can be a general-purpose processor, such as a CPU; an integrated system, such as a system-on-a-chip (SoC); an embedded control core, such as a microcontroller unit (MCU); a dedicated signal processing unit, such as a digital signal processor (DSP); a graphics rendering core, such as a graphics processing unit (GPU); a programmable logic device, such as an application-specific integrated circuit (ASIC); a field-programmable gate array (FPGA); or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components.

[0331] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the embodiments of this application, please refer to the descriptions of the method embodiments in this application for understanding.

[0332] Example 3

[0333] This application provides a computing device 50, with reference to... Figure 4 The computing device 50 includes: a storage unit 51, a communication bus 52, and a processing unit 53, wherein:

[0334] The storage component 51 is used to store the gesture interaction method program;

[0335] The communication bus 52 is used to realize the connection and communication between the storage component 51 and the processing component 53.

[0336] The processing unit 53 is used to execute the gesture interaction method program to implement the steps of the method described in Embodiment 1.

[0337] The type or structure of the storage component 51 can be found in the storage medium section below, and will not be repeated here.

[0338] The processing unit 53 can be a general-purpose processor, such as a CPU; an integrated system, such as a system-on-a-chip (SoC); an embedded control core, such as a microcontroller unit (MCU); a dedicated signal processing unit, such as a digital signal processor (DSP); a graphics rendering core, such as a graphics processing unit (GPU); a programmable logic device, such as an application-specific integrated circuit (ASIC); a field-programmable gate array (FPGA); or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components.

[0339] In some embodiments, the computing device 50 may further include an input device 54, an output device 55, and an external communication interface 56, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0340] In some embodiments, input device 54 may include, for example, a keyboard, mouse, microphone, etc. Output device 55 may output various information to the outside, including displays, speakers, printers, projectors, communication networks and their connected remote output devices, etc. External communication interface 56 may be wired.

[0341] Examples include standard serial ports (RS232), general-purpose interface buses (GPIB), Ethernet, and universal serial buses (USB). Wireless interfaces such as WiFi and Bluetooth are also possible.

[0342] The description of the above-described 50 embodiments of the computing device is similar to that of the above-described method embodiments, and has similar beneficial effects. For technical details not disclosed in the embodiments of this application, please refer to the description of the method embodiments in this application for understanding.

[0343] Example 4

[0344] This application provides a computer-readable storage medium storing an executable program, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0345] For example, a computer-readable storage medium may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A computer-readable storage medium is a tangible device capable of holding and storing instructions used by an instruction execution device. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include:

[0346] Portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), flash memory, portable compact disc read-only memory (CD-ROM), digital versatile discs (DVDs), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combinations thereof. Among them:

[0347] The RAM includes: Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).

[0348] The ROM includes: Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM).

[0349] The description of the computer-readable storage medium embodiments above is similar to the description of the method embodiments above, and has similar beneficial effects. For technical details not disclosed in the embodiments of this application, please refer to the description of the method embodiments in this application for understanding.

[0350] Understandably, without conflict, the technical features in the technical solutions described in each embodiment can be arbitrarily combined to form new embodiments. For example, each structure in each embodiment can be implemented as an independent embodiment, and the structures can be arbitrarily combined; some or all of the structures in different embodiments can be arbitrarily combined. Each step in each embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined; the order of the steps can be arbitrarily interchanged; some or all of the steps in different embodiments can be arbitrarily combined.

[0351] In this document, when the terms "embodiment," "implementation," or "example" are used, it means that the specific features described in connection with these implementations or examples are included in at least one implementation, embodiment, or example of this application. It should be noted that the illustrative expressions of the above terms do not necessarily refer to the same implementation, embodiment, or example. Furthermore, the specific features described, such as structures or steps, can be appropriately combined in any one or more implementations, embodiments, or examples.

[0352] In some embodiments, prefixes such as "first" and "second" are used merely to distinguish different descriptive objects and do not impose restrictions on the position, order, priority, or value of the descriptive objects. The description of the descriptive objects is given in the context of the embodiments, and the use of prefixes does not constitute unnecessary restrictions. For example, the numerical value of a descriptive object is not limited by ordinal numbers and can be one or more. Taking "first device" as an example, the numerical value of "device" can be one or more. Furthermore, objects modified by different prefixes can be the same or different. For example, if the descriptive object is "device," then "first device" and "second device" can be the same device or different devices, and their types can be the same or different. Describing "first" does not necessarily imply the existence of "second," and discussing "second" does not necessarily imply the existence of "first."

[0353] In some embodiments, unless otherwise stated, elements expressed in the singular form, such as "a," "the," "the," "the," "the," "the," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular or a plural expression. In some embodiments, "multiple" refers to two or more.

[0354] In some embodiments, specific operational steps, such as flowcharts, are provided. However, it should be noted that these operational steps may be added or removed based on conventional or non-creative effort. The order of steps listed in the embodiments is only one of many possible orders and does not represent the only order. When executed in actual devices, systems, or server products, the steps can be executed either in the order shown in the embodiments or the accompanying drawings, or in parallel in a parallel processor or multi-threaded processing environment.

[0355] The embodiments of this application may be methods, apparatus (systems), and / or computer-readable storage media. The computer-readable storage medium may carry an executable program for causing a processor to implement various aspects of this application. The executable program may be program code written in any combination of one or more programming languages ​​for executing the embodiments of this application. Programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​or other programming languages ​​such as "C". The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer. The network may be a wired network or a wireless network.

[0356] In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of an executable program. These electronic circuits can execute executable programs to implement various aspects of this application.

[0357] The executable program described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the executable program from the network and forwards it for storage on a computer-readable storage medium within the respective computing / processing device.

[0358] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and / or computer-readable storage media according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by an executable program.

[0359] These executable programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These executable programs can also be stored in a computer-readable storage medium containing instructions that cause a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable storage medium storing the instructions comprises an article of manufacture including instructions that implement aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram. The executable programs can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable data processing apparatus, or other device, implement the functions / actions specified in one or more blocks of the flowchart and / or block diagram. In some embodiments, the disclosed apparatus and methods can be implemented in a variety of other ways. The described device embodiments are for illustrative purposes only. For example, the module division represents only one logical functional division method. In actual implementation, multiple modules or components may be combined or integrated into another system, or certain features may be ignored or specific operations may not be performed. The coupling, direct coupling, or communication connection between the components can be achieved indirectly through interfaces, devices, or modules. The connection form can be electrical, mechanical, or other types.

[0360] In some embodiments, the modules described as separate components may or may not be physically separate; the components shown as modules may or may not be physical modules; these modules may or may not be concentrated in one place or distributed across multiple network modules. In practical applications, some or all of the modules can be selected to achieve the objectives of this embodiment, depending on the requirements.

[0361] In some embodiments, the integration of functional modules is flexible and diverse: they can all be integrated into one processing module, each can be an independent module, or two or more functional modules can be integrated into one module. These integrated modules can be implemented in pure hardware or in a combination of hardware and software functional modules.

[0362] In some embodiments, all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The relevant program can be stored in a computer-readable storage medium, such as ROM, RAM, magnetic disk, or optical disk, and implements the steps of the above method embodiments when executed. If the integrated modules of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Therefore, the technical solutions of the embodiments of this application, in essence or contributing to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and contains several instructions for causing an electronic device (such as a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various embodiments of this application. Therefore, the embodiments of this application are not limited to any specific hardware and software combination.

[0363] It should be understood that the above embodiments are exemplary and are not intended to encompass all possible implementations of the technical solutions of this application. Various modifications and changes can be made to the above embodiments without departing from the scope of this application. The above embodiments merely illustrate several implementations of this application and do not limit the scope of protection of this patent application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A gesture interaction method, characterized in that, The method includes: From the raw video stream captured by the camera, a first image containing the operator is obtained, and the human body reference length of the operator is identified. The pixel length of the human body reference length in the first image is measured. The human body reference length is a linear distance with fixed physical dimensions on the operator's body. Based on the first image, a second image containing the operator's hand is determined, and the coordinates of key points of the hand in the second image are obtained; Gesture interaction parameters are determined based on the pixel length; wherein, the gesture interaction parameters are adjusted as the pixel length changes: when the operator moves away from the camera and the pixel length decreases, the gesture interaction parameters are increased; when the operator moves closer to the camera and the pixel length increases, the gesture interaction parameters are decreased. Based on the coordinates of the hand key points in the second image and the gesture interaction parameters, the intention of the interaction operation is determined to generate control instructions for the gesture interaction operation.

2. The gesture interaction method according to claim 1, characterized in that, The human body reference length is the operator's shoulder width; Measuring the pixel length of the human body reference length in the first image includes: identifying the coordinates of the operator's left and right shoulder peaks in the first image, and calculating the pixel distance between the two points as the pixel length.

3. The gesture interaction method according to claim 1, characterized in that, The gesture interaction parameter is the gesture mapping ratio; The step of determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, in order to generate control instructions for the gesture interaction operation, includes: The coordinates of the key hand points in the second image are converted into operation positions on the screen using the gesture mapping ratio. The intention of the interactive operation is determined based on the operation position on the screen, so as to generate control instructions for the gesture interaction operation.

4. The gesture interaction method according to claim 1, characterized in that, The gesture interaction parameter is the gesture trigger threshold; The step of determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, in order to generate control instructions for the gesture interaction operation, includes: Calculate the hand posture features or hand movement distance based on the coordinates of the key hand points in the second image; The hand posture features or hand movement distance are compared with the gesture trigger threshold. When the gesture trigger threshold is exceeded, it is identified as a valid gesture action, and a corresponding control command for gesture interaction is generated.

5. The gesture interaction method according to claim 1, characterized in that, The step of determining the gesture interaction parameters based on the pixel length includes: Establish the correspondence between the physical area corresponding to the operator's body reference length and the screen display area; This allows the hand to move within the human body's baseline length range, corresponding to the operation displacement of the entire screen area or a portion of the screen area.

6. The gesture interaction method according to claim 1, characterized in that, The step of obtaining the coordinates of key hand points in the second image includes: Identify the operator's hand orientation and determine whether the operator's hand is left-handed or right-handed; Calculate the coordinates of the hand root in the second image, and establish a local coordinate system for the hand using the hand root as an anchor point; Based on the aforementioned local hand coordinate system, the coordinates of other key points of the hand, excluding the base of the hand, are extracted.

7. The gesture interaction method according to claim 1, characterized in that, The step of obtaining the coordinates of key hand points in the second image includes a jump compensation step: Get the original coordinates of the hand in the current frame; The predicted hand coordinates of the current frame are calculated using a prediction algorithm based on the motion trends of the previous N frames; where N is an integer greater than or equal to 2. The original hand coordinates are compared with the predicted hand coordinates. If the deviation between the two exceeds a preset human motion threshold, the current frame is determined to be abnormal, and the predicted hand coordinates are used as the coordinates of the hand key points in the second image. If the deviation is within the normal range, the original hand coordinates are used, and the internal parameters of the prediction algorithm are updated using the original hand coordinates.

8. The gesture interaction method according to claim 1, characterized in that, After determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters to generate control instructions for the gesture interaction operation, a smoothing process step is also included: The adaptive filter cutoff frequency is dynamically adjusted based on the moving speed of the key hand points: when the moving speed is greater than a preset speed threshold, the filter cutoff frequency is increased to reduce signal delay. When the moving speed is less than or equal to the preset speed threshold, the filter cutoff frequency is reduced to suppress signal jitter.

9. The gesture interaction method according to claim 1, characterized in that, When generating control commands for gesture interaction, a step to prevent accidental touches is also included: Monitor changes in the state of hand gestures, and enter a cooldown period after a valid trigger command is detected; During the cooling period, subsequent identical trigger signals are ignored to prevent repeated execution of commands due to hand tremors.

10. The gesture interaction method according to claim 1, characterized in that, Before determining the intent of the interaction based on the coordinates of the hand key points in the second image and the gesture interaction parameters, an orientation calibration step is also included: Identify the operator's body orientation angle; Based on the body's orientation angle, adjust the starting point or direction of the operation area on the screen display area so that when the operator stands sideways, the direction of hand movement is consistent with the operation direction on the screen display area. The determination of the intent of the interactive operation specifically includes: determining the intent of the interactive operation based on the adjusted starting point or direction of the operation area.

11. The gesture interaction method according to claim 1, characterized in that, Before determining the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, a gaze calibration step is also included: Identify the operator's eye movement trajectory; By combining the human eye gaze trajectory with the human body reference length, a two-layer calibration mechanism is constructed, wherein: the first layer determines the overall mapping ratio based on the human body reference length, and the second layer determines the operation area based on the human eye gaze trajectory; The determination of the intent of the interactive operation specifically includes: determining the intent of the interactive operation based on the overall mapping ratio and operation area determined by the dual-layer calibration mechanism.

12. The gesture interaction method according to claim 1, characterized in that, The step of determining a second image containing the operator's hand based on the first image includes an image enhancement step: Extract a third image containing the hand from the first image; The third image is subjected to detail magnification and enhancement processing to obtain an enhanced image with higher clarity than the original video stream, and the enhanced image is used as the second image; Based on the second image, key points of the hand are identified.

13. The gesture interaction method according to claim 12, characterized in that, The detailed magnification and enhancement processing includes: The pixels of the third image are reconstructed using a weighted interpolation algorithm; A window function is used to limit the range of neighboring pixels involved in the calculation, and smoothing weights are assigned to pixels at different locations to suppress artifacts at image edges.

14. The gesture interaction method according to claim 1, characterized in that, Before acquiring the first image containing the operator from the raw video stream captured from the camera, an operator locking step is also included: Track multiple target objects in the original video stream and assign them unique identifiers; Calculate the overlap between each target object and the preset interaction area, the dwell time, and the human body orientation score to obtain the interaction intent score; Select the target object with the highest interaction intent score as the operator.

15. The gesture interaction method according to claim 1, characterized in that, The step of obtaining the coordinates of key hand points in the second image also includes a hand orientation correction step: Obtain the angle between the vertical axis of the palm and the vertical axis of the second image; wherein, the vertical axis of the palm is the direction extending from the wrist to the fingers; The image region containing the hand is rotated and corrected according to the included angle, so that the longitudinal direction of the palm in the corrected hand image is consistent with the preset standard direction, so as to accurately extract the key points of the hand.

16. The gesture interaction method according to claim 1, characterized in that, The key points of the hand include the wrist, palm, and the joints of each finger. When determining the intent of the interactive operation to generate control instructions for the gesture interactive operation, at least one of the following hand state features is also utilized: the degree of finger flexion, used to characterize the opening and closing state of the fingers. The direction the palm faces is used to indicate the spatial orientation of the hand; The number of fingers extended is used to represent the number of fingers that can be straightened. And the relative instantaneous velocity of each finger joint, used to characterize the speed of hand movement.

17. The gesture interaction method according to claim 7, characterized in that, The prediction algorithm is the Kalman filter algorithm; The step of using a prediction algorithm to calculate the predicted hand coordinates for the current frame based on the motion trends of the previous N frames includes: Based on the average velocity and acceleration of hand movement in historical frames, the global displacement of the hand is calculated in time series to obtain the predicted coordinates of the hand in the current frame.

18. The gesture interaction method according to claim 8, characterized in that, The adaptive filter is a frequency conversion low-pass filter.

19. The gesture interaction method according to claim 9, characterized in that, The cooling time is 300 milliseconds to 800 milliseconds.

20. A gesture interaction device, characterized in that, The device includes: The measurement module is used to acquire a first image containing the operator from the raw video stream captured by the camera, identify the operator's human body reference length, and measure the pixel length of the human body reference length in the first image; wherein, the human body reference length is a linear distance with fixed physical dimensions on the operator's body; The acquisition module is used to determine a second image containing the operator's hand based on the first image, and to acquire the coordinates of key points of the hand in the second image; The first determining module is used to determine gesture interaction parameters based on the pixel length; wherein the gesture interaction parameters are adjusted as the pixel length changes: when the operator moves away from the camera and the pixel length decreases, the gesture interaction parameters are increased; when the operator moves closer to the camera and the pixel length increases, the gesture interaction parameters are decreased. The second determining module is used to determine the intent of the interaction operation based on the coordinates of the hand key points in the second image and the gesture interaction parameters, so as to generate control instructions for the gesture interaction operation.

21. A computing device, characterized in that, The computing device includes: a storage component, a communication bus, and a processing component, wherein: The storage component is used to store gesture interaction method programs; The communication bus is used to enable communication between the storage component and the processing component; The processing unit is configured to execute a gesture interaction method program to implement the steps of the method as described in any one of claims 1 to 19.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an executable program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 19.

Images