A gesture recognition based method and system for controlling a tablet computer in midair

Abstract

The application belongs to the technical field of human-computer interaction, and discloses a tablet computer remote control method and system based on gesture recognition, which comprises the following steps: acquiring the continuous interaction state of the user's hand in the interaction space in front of the tablet computer, determining the intention of the continuous interaction state according to a preset control intention confirmation rule, and establishing the corresponding control reference information after confirming that the remote control mode is entered; based on the control reference information, the continuous interaction state is calibrated for the control behavior, the influence of the user's active control and the change of the holding posture is separated, and the calibrated control information representing the user's active control behavior is obtained; the spatial mapping relationship between the interaction space and the display interface of the tablet computer is established, and the current control object is determined according to the calibrated control information; and the accuracy and stability of the remote control of the tablet computer are improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically, to a method and system for controlling a tablet computer with air gestures based on gesture recognition. Background Technology

[0002] With the development of smart terminal interaction technology, mobile devices such as tablets are gradually evolving from traditional touch control to more natural and convenient human-computer interaction methods. Among them, gesture recognition-based air control technology can use cameras or sensors to collect users' hand movements, allowing users to complete some operations without directly touching the screen, providing a new way to improve the mobile terminal interaction experience.

[0003] However, existing methods for controlling tablets with air gestures typically rely on hand postures or simple movements captured at a specific moment for identification, triggering corresponding operations based on preset rules. Due to the lack of comprehensive analysis of the user's continuous behavioral state, the system struggles to accurately determine whether the user has a genuine intention to control the device. It is easily affected by factors such as natural hand movements, unconscious actions, and environmental changes, leading to false triggering or misidentification.

[0004] Meanwhile, when using a tablet in actual use, changes in grip posture, position, and angle of use will cause a relative shift in the hand's position if the device is held in hand. Existing technologies often struggle to distinguish between the effective movements of the user's active operation and the overall shift caused by changes in device posture, easily leading to problems such as inaccurate control positioning and decreased recognition stability. Furthermore, existing solutions do not adequately consider the correspondence between the hand interaction space and the display interface, making it difficult to accurately determine the screen operation object corresponding to the gesture in multi-target interaction scenarios.

[0005] Furthermore, existing gesture recognition methods mostly rely on static posture features for matching. For gestures that are similar in shape but different in meaning, recognition ambiguity is likely to occur. The recognition ability for continuous and compound actions is limited, making it difficult to meet the application needs in complex interaction scenarios.

[0006] In addition, existing air gesture control methods usually lack the ability to judge the completeness and continuous effectiveness of gesture execution. When the user's action is not completed, the target posture is briefly displayed, or an unexpected deviation occurs, the system may execute the control command in advance, resulting in erroneous operation and reducing the reliability and controllability of the interaction process.

[0007] In view of this, the present invention proposes a method and system for air control of a tablet computer based on gesture recognition to solve the above problems. Summary of the Invention

[0008] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a method for air-controlling a tablet computer based on gesture recognition, comprising: S1. Obtain the continuous interaction state of the user's hand in the interaction space in front of the tablet computer, determine the intention of the continuous interaction state according to the preset control intention confirmation rule, and establish the corresponding control reference information after confirming that the air control mode has been entered. S2. Based on the control reference information, calibrate the control behavior in the continuous interaction state, separate the influence of user active control and changes in holding posture, and obtain calibration control information that characterizes the user's active control behavior; establish the spatial mapping relationship between the interaction space and the tablet computer display interface, and determine the current control object based on the calibration control information. S3. Combine calibration control information and the current control object to construct a gesture shape descriptor. Based on the correspondence between the temporal change trajectory of the gesture shape descriptor and the preset interactive gesture, perform context disambiguation processing to determine the target interactive gesture corresponding to the current interaction process and generate control instructions. S4. Within the preset gesture integrity determination window, continuously monitor whether the gesture shape descriptor remains within the tolerance range of the target interactive gesture. If it remains within the tolerance range, execute the interactive operation according to the control command. If it deviates, determine that the current interactive process has ended and close the air control mode.

[0009] Preferably, the method for obtaining the continuous interaction state includes: The tablet's front-facing camera continuously captures images of the user's hand in the interactive space in front of the display screen. The hand in each frame is tracked and identified, and the corresponding key point coordinates are extracted. A sequence of hand movement changes is established based on the key point coordinates of each consecutive frame, which serves as a continuous interactive state representing the user's hand position and gesture changes.

[0010] Preferably, the method for obtaining the control reference information includes: The overall hand motion velocity vector is calculated based on the continuous interaction state, and the intention is determined by combining the preset control intention confirmation rules; the preset control intention confirmation rules include a preset static intention threshold and a preset intention confirmation frame number. The magnitude of the overall hand movement velocity vector is continuously compared with a preset static intention threshold. When the magnitude is continuously lower than the preset static intention threshold and the preset intention confirmation frame number is reached, it is determined that the user has the intention to enter the air control mode. The coordinates of the palm center point and the index finger tip corresponding to the user's hand image in the current confirmation frame are read. A control reference point pair is constructed based on the palm center coordinates and the index finger tip coordinates as the control reference information to activate the air control mode.

[0011] Preferably, the method for obtaining calibration control information characterizing the user's active control behavior includes: The coordinates of the current palm center point and the current index fingertip are obtained based on the continuous interaction state, and the overall displacement vector and the comprehensive displacement vector are calculated relative to the control reference information. The comprehensive displacement vector is calibrated based on the overall displacement vector to eliminate the influence of the overall displacement caused by the change of holding posture, and the relative displacement vector of the index fingertip coordinates relative to the palm center point is obtained. The relative displacement vector is used as the calibration control information.

[0012] Preferably, the method for determining the current manipulated object includes: Based on the control reference information, determine the initial hand interaction posture when entering the air control mode, and establish the correspondence between the initial hand interaction posture and the preset initial screen position to form the spatial mapping starting point between the interaction space and the tablet computer display interface; establish the displacement mapping rules between the interaction space displacement and the screen displacement based on the spatial mapping starting point. Based on the calibration control information, determine the user's active control displacement relative to the initial interaction posture of the hand, and obtain the corresponding current screen position according to the displacement mapping rule; obtain the preset response area corresponding to each interactive interface element in the current display interface, and determine the interactive interface element corresponding to the preset response area into which the current screen position falls as the current control object.

[0013] Preferably, the method for constructing a gesture morphology descriptor includes: The relative displacement direction and magnitude of the index finger relative to the center point of the palm are determined based on the calibration control information, and the bending state, finger opening degree, angle between adjacent fingers and the direction of the index finger are calculated based on the coordinates of key points on the user's hand. Read the interaction attributes corresponding to the currently manipulated object, and determine the target shape parameters to describe the interaction characteristics of the currently manipulated object based on the interaction attributes; combine the target shape parameters with the relative displacement direction and relative displacement amplitude of the index finger to form the gesture shape descriptor corresponding to the current interaction process.

[0014] Preferably, the method for generating control commands includes: Continuously acquire gesture shape descriptors corresponding to a series of consecutive image frames, and assemble the temporal change trajectory of the gesture shape descriptors in chronological order; read the set of preset interactive gestures corresponding to the current manipulated object, and establish the correspondence between each preset interactive gesture and the corresponding standard temporal change trajectory; The temporal change trajectory of the gesture shape descriptor is matched with each standard temporal change trajectory. Based on the matching results, interactive gestures that do not correspond to the interaction attributes of the current controlled object are filtered out. Context disambiguation is performed on the remaining interactive gestures to determine the target interactive gesture corresponding to the current interaction process. Read the pre-established instruction mapping relationship between the target interactive gesture and the currently manipulated object, and generate the corresponding control instructions based on the instruction mapping relationship.

[0015] Preferably, the method for setting the preset gesture integrity determination window includes: Based on the standard temporal change trajectory corresponding to the target interactive gesture, determine the starting form, ending form, and duration range of the target interactive gesture; based on the allowable fluctuation range of the gesture form descriptor corresponding to each stage of the target interactive gesture, determine the form tolerance range corresponding to each stage; associate the duration range with the form tolerance range corresponding to each stage to construct a preset gesture integrity judgment window for the target interactive gesture.

[0016] Preferably, the method for continuously monitoring whether the gesture shape descriptor remains within the tolerance range of the target interactive gesture includes: Within the preset gesture integrity determination window, continuously acquire the gesture shape descriptor corresponding to the user's hand image in the current frame, and read the shape tolerance range of the target interaction gesture corresponding to the time sequence stage according to the current gesture shape descriptor. The current gesture shape descriptor is compared with the shape tolerance range of the corresponding stage, and the duration of the current target interaction gesture is simultaneously counted to determine whether the current gesture shape descriptor is continuously maintained within the shape tolerance range of the corresponding stage and the duration is within the duration range. When the current gesture descriptor remains within the shape tolerance range of the corresponding stage and the duration is within the duration range, the target interaction gesture is determined to be fully executed, and the corresponding interaction operation is executed according to the control command. When the current gesture descriptor deviates from the shape tolerance range of the corresponding stage, or the duration exceeds the duration range, the current interaction process is determined to end, the air control mode is turned off, and the gesture descriptor, target interaction gesture and control command corresponding to the current interaction process are cleared.

[0017] A gesture recognition-based air control system for tablet computers includes: The intent confirmation module is used to obtain the continuous interaction state of the user's hand in the interaction space in front of the tablet computer, determine the intent of the continuous interaction state according to the preset control intent confirmation rules, and establish corresponding control benchmark reference information after confirming entry into the air control mode. The behavior calibration module calibrates the control behavior in continuous interactive states based on the control reference information, separates the influence of user active control and changes in holding posture, and obtains calibrated control information that characterizes the user's active control behavior; based on the calibrated control information, it establishes a spatial mapping relationship with the tablet computer display interface to determine the current control object. The semantic parsing module combines calibration control information and the current control object to construct a gesture shape descriptor. Based on the temporal change trajectory of the gesture shape descriptor and the correspondence between the preset interactive gestures, it performs context disambiguation to determine the target interactive gesture corresponding to the current interaction process and generate control instructions. The execution control module continuously monitors whether the gesture shape descriptor remains within the tolerance range of the target interactive gesture within the preset gesture integrity judgment window. If it remains within the tolerance range, the interactive operation is executed according to the control command. If it deviates, the current interactive process is determined to end and the air control mode is turned off.

[0018] Compared with the prior art, the present invention has the following beneficial effects: By continuously sensing and comprehensively analyzing the user's hand interaction state, the tablet computer can understand the user's operation intention and complete the interactive control without directly touching the screen. Compared with the existing air interaction methods based on simple gesture triggers, this invention does not only judge based on the hand posture at a single moment, but also establishes a control reference benchmark by combining the user's continuous behavior state before and after entering the air control mode. This allows the system to perceive the user's transition from a natural state to an active operation state, thereby reducing misidentification caused by unconscious hand movements, environmental interference, or accidental actions, and improving the reliability of air control.

[0019] By introducing control reference information to dynamically calibrate the user interaction process, the system can distinguish between effective movements generated by user active control and overall offsets caused by factors such as changes in device grip and usage posture. By establishing a correspondence between the user's hand interaction space and the display interface, air gestures can be mapped to specific screen operation objects, achieving an effective conversion from spatial actions to interface interaction.

[0020] By using gesture morphology descriptors to comprehensively represent the spatial morphology of gestures and their changes over time, and combining contextual information to distinguish similar gestures, the system can understand the meaning of user operations based on the complete action process, rather than relying on a single fixed posture to make a judgment. This can effectively improve the problems that are easy to occur in existing gesture recognition schemes, such as homomorphism, action interruption, and difficulty in distinguishing continuous operations, and improve the recognition accuracy and interaction naturalness in complex interaction scenarios.

[0021] By performing a completeness check on the gesture execution process, the control command will only be executed when the gesture state meets the preset conditions, and can exit the current interaction state in a timely manner when the user's action ends or there is a significant deviation. This can avoid the execution of erroneous commands caused by short actions, incomplete actions or accidental movements, and improve the safety and controllability of the system control process. Attached Figure Description

[0022] Figure 1This is a schematic diagram of a gesture recognition-based air control method for a tablet computer according to the present invention. Figure 2 This is a schematic diagram of a gesture recognition-based air control system for a tablet computer according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1 Please see Figure 1 As shown, this embodiment provides a method for air control of a tablet computer based on gesture recognition, specifically including the following steps: S1. Obtain the continuous interaction state of the user's hand in the interaction space in front of the tablet computer, determine the intention of the continuous interaction state according to the preset control intention confirmation rule, and establish the corresponding control reference information after confirming that the air control mode has been entered. S2. Based on the control reference information, calibrate the control behavior in the continuous interaction state, separate the influence of user active control and changes in holding posture, and obtain calibration control information that characterizes the user's active control behavior; establish the spatial mapping relationship between the interaction space and the tablet computer display interface, and determine the current control object based on the calibration control information. S3. Combine calibration control information and the current control object to construct a gesture shape descriptor. Based on the correspondence between the temporal change trajectory of the gesture shape descriptor and the preset interactive gesture, perform context disambiguation processing to determine the target interactive gesture corresponding to the current interaction process and generate control instructions. S4. Within the preset gesture integrity determination window, continuously monitor whether the gesture shape descriptor remains within the tolerance range of the target interactive gesture. If it remains within the tolerance range, execute the interactive operation according to the control command. If it deviates, determine that the current interactive process has ended and close the air control mode.

[0025] Methods for obtaining continuous interaction states include: The tablet's front-facing camera continuously captures images of the user's hand in the interactive space in front of the display screen. The hand in each frame is tracked and identified, and the corresponding key point coordinates are extracted. A sequence of hand movement changes is established based on the key point coordinates of each consecutive frame, which serves as a continuous interactive state representing the user's hand position and gesture changes.

[0026] Specifically: The front-facing camera of the tablet continuously captures images of the user's hand in the interactive space in front of the tablet screen, forming a sequence of hand images; the size of the interactive space depends on the maximum field of view of the front-facing camera; the user's hand appears in each frame of the hand image sequence, and the coordinates of the key points corresponding to the user's hand are extracted by identifying the key points of the hand skeleton; In this embodiment, at least 21 key point coordinates of the user's hand need to be extracted for each frame of the user's hand image, including the center point of the palm, the base joint of the thumb, the first joint of the thumb, the second joint of the thumb, the fingertip of the thumb, and the base joint, first joint, second joint, and fingertip of the index, middle, ring, and little fingers respectively. The obtained 21 key point coordinates are represented by two-dimensional pixel coordinates in the image coordinate system, which constitutes the key point coordinate set of the current user's hand image. The key point coordinate sets corresponding to all user hand images are collected to form a hand movement change sequence, which serves as a continuous interactive state representing the user's hand position and gesture changes.

[0027] Methods for obtaining control reference information include: The overall hand motion velocity vector is calculated based on the continuous interaction state, and the intention is determined by combining the preset control intention confirmation rules; the preset control intention confirmation rules include a preset static intention threshold and a preset intention confirmation frame number. The magnitude of the overall hand movement velocity vector is continuously compared with a preset static intention threshold. When the magnitude is continuously lower than the preset static intention threshold and the preset intention confirmation frame number is reached, it is determined that the user has the intention to enter the air control mode. The coordinates of the palm center point and the index finger tip corresponding to the user's hand image in the current confirmation frame are read. A control reference point pair is constructed based on the palm center coordinates and the index finger tip coordinates as the control reference information to activate the air control mode.

[0028] Specifically: After acquiring the continuous interaction state, the key point coordinate set of two consecutive frames of user hand images is read in chronological order. The difference in pixel coordinates of the palm center point between two adjacent frames of user hand images is divided by the inter-frame time interval to obtain the inter-frame velocity vector of the palm center point. The inter-frame velocity vector of the palm center point is used as the overall hand motion velocity vector. The overall hand motion velocity vectors corresponding to all adjacent user hand image frames in the hand image sequence are counted to form a hand motion change sequence. Subsequently, the intent is determined for continuous interaction states according to the preset control intent confirmation rules. In this embodiment, the preset control intent confirmation rules include a preset static intent threshold and intent confirmation frame count. The preset static intent threshold is calibrated according to the resolution and frame rate of the tablet's front camera and is used to distinguish between the user's active static state and the unconscious hand movement state. The preset intent confirmation frame count is used to limit the minimum duration for which the user's hand is in an active static state. The magnitude of the overall hand motion velocity vector corresponding to each frame of the user's hand image is calculated and compared with the preset static intention threshold in chronological order. When the magnitude of the overall hand motion velocity vector is continuously lower than the preset static intention threshold and the cumulative number reaches the preset intention confirmation frame number, it is determined that the user's hand has changed from a free movement state to a stable stationary state and has the intention to enter the air control mode.

[0029] After confirming the user's intention to enter the air control mode, the coordinates of the palm center point and the index finger tip corresponding to the user's hand image in the current confirmation frame are read. Among them, when the magnitude of the overall hand motion velocity vector is continuously lower than the preset static intention threshold and a preset intention confirmation frame number is reached, the image frame corresponding to the preset intention confirmation frame number is used as the current confirmation frame user hand image. The palm center point coordinates are used to represent the position reference of the current overall hand posture, and the index finger tip coordinates are used to represent the position reference of the user's subsequent fine control actions. Furthermore, a control reference point pair is established using the coordinates of the center point of the palm and the index fingertip. This control reference point pair is saved as control reference information to describe the initial spatial relationship between the overall hand posture and the fingertip position when the user enters the air control mode. The control reference point pair includes not only the coordinates of the center point of the palm and the index fingertip, but also at least one of the relative displacement relationship, spatial distance, and directional relationship between the two. For example, when entering the air control mode, if the coordinates of the center point of the palm in the current image frame are (320, 280) and the coordinates of the index fingertip are (365, 225), then a corresponding control reference point pair is established, and the relative displacement vector between them is recorded as Δ = (45, -55). During subsequent air control operations, by calculating the relative displacement of the current index fingertip coordinates relative to the current palm center coordinates in real time and comparing it with the relative displacement relationship in the control reference point pair, the overall displacement caused by the overall movement of the tablet and the relative displacement generated by the user's active movement of the index finger can be separated, thus obtaining more accurate active control information.

[0030] Methods for obtaining calibration control information that characterizes user-initiated control behavior include: The coordinates of the current palm center point and the current index fingertip are obtained based on the continuous interaction state, and the overall displacement vector and the comprehensive displacement vector are calculated relative to the control reference information. The comprehensive displacement vector is calibrated based on the overall displacement vector to eliminate the influence of the overall displacement caused by the change of holding posture, and the relative displacement vector of the index fingertip coordinates relative to the palm center point is obtained. The relative displacement vector is used as the calibration control information.

[0031] Specifically: After entering the air control mode, the system continuously receives the continuous interaction state corresponding to the user's hand image, and traverses each frame of the user's hand image that makes up the continuous interaction state in chronological order, and extracts the coordinates of the center point of the palm and the fingertip of the index finger corresponding to each frame. For each frame of the user's hand image, the displacement change of the corresponding palm center point coordinates relative to the palm center point coordinates in the control reference information is calculated to form an overall displacement vector; the displacement change of the corresponding index fingertip coordinates relative to the index fingertip coordinates in the control reference information is calculated to form a comprehensive displacement vector. Among them, the overall displacement vector is used to characterize the overall spatial displacement caused by changes in the user's holding posture or the overall movement of the tablet computer; while the displacement change of the index fingertip includes both the positional change caused by the overall spatial displacement and the relative motion change caused by the user's active air control action. Therefore, the displacement change of the index fingertip is a comprehensive displacement vector formed by the combined action of the overall spatial displacement motion and the user's active control motion.

[0032] In order to obtain the user's true active control behavior, this embodiment uses the overall displacement vector as the overall motion compensation amount to correct the comprehensive displacement vector, eliminate the influence of the overall spatial displacement on the movement trajectory of the index fingertip, and obtain the relative displacement vector of the index fingertip relative to the center point of the palm. The relative displacement vector is used as calibration control information that only represents the user's active control behavior.

[0033] It should be noted that when a user holds a tablet, the device may sway slightly with the user's posture. If the index fingertip coordinates are used directly for control, the change in holding posture may be easily misidentified as a control action. Therefore, this embodiment introduces the center point of the palm as an overall motion reference and uses the control reference information to establish a reference spatial relationship between the center point of the palm and the index fingertip. In subsequent continuous interaction, the relative motion relationship between the two is calculated in real time, thereby effectively separating the overall displacement caused by the change in holding posture from the local displacement caused by the user's active control, and improving the stability of air control positioning.

[0034] Methods for determining the currently manipulated object include: Based on the control reference information, determine the initial hand interaction posture when entering the air control mode, and establish the correspondence between the initial hand interaction posture and the preset initial screen position to form the spatial mapping starting point between the interaction space and the tablet computer display interface; establish the displacement mapping rules between the interaction space displacement and the screen displacement based on the spatial mapping starting point. Based on the calibration control information, determine the user's active control displacement relative to the initial interaction posture of the hand, and obtain the corresponding current screen position according to the displacement mapping rule; obtain the preset response area corresponding to each interactive interface element in the current display interface, and determine the interactive interface element corresponding to the preset response area into which the current screen position falls as the current control object.

[0035] Specifically: When entering the air control mode, the system reads the control reference information and uses the initial hand interaction posture corresponding to the control reference information when entering the air control mode as the reference starting point of the interaction space; wherein, the initial hand interaction posture is determined by the control reference point pair, which is used to characterize the initial spatial relationship between the center point of the palm and the fingertip of the index finger when the user starts air control.

[0036] At the same time, the system determines the initial screen position corresponding to the initial hand interaction posture. The initial screen position is used to represent the initial mapping position of the interaction space in the tablet computer display interface. In this embodiment, it is set to the center position of the display interface. The correspondence between the initial hand interaction posture and the initial screen position is defined as the starting point of the spatial mapping between the interaction space and the tablet computer display interface. The starting point of the spatial mapping indicates that when the user's hand is in the initial interaction posture, the virtual cursor is located at the coordinates of the center point of the screen.

[0037] Subsequently, based on the display range of the tablet computer's display interface, a displacement mapping rule is established between the displacement in the interactive space and the screen displacement. The displacement mapping rule is used to describe the amount of displacement change on the screen corresponding to the unit displacement of the user's hand in the interactive space, so that the active control action in the interactive space can be converted into a position change on the display interface. In this embodiment, the system pre-configures the corresponding screen displacement for the unit displacement in the interactive space according to the effective control range of the interactive space and the display range of the tablet computer's display interface, so that the movement of the user's hand in the interactive space can be proportionally converted into a position change on the screen. For example, when the effective horizontal control range of the interactive space is 150mm, and the horizontal display range of the tablet computer's screen is 1200 pixels, a horizontal displacement mapping relationship of 1mm corresponding to 8 pixels can be established; when the effective vertical control range of the interactive space is 100mm, and the vertical display range of the screen is 800 pixels, a vertical displacement mapping relationship of 1mm corresponding to 8 pixels can be established. Subsequently, after obtaining the user's active control displacement based on the calibration control information, the corresponding screen displacement is calculated according to the above horizontal and vertical displacement mapping relationships, thereby updating the current screen position.

[0038] Subsequently, the calibration control information is read, and the active control displacement of the user relative to the initial interaction posture of the hand is determined based on the calibration control information. Then, according to the displacement mapping rule, the active control displacement is converted into the corresponding screen displacement. Then, with the initial screen position as the reference, the screen displacement is superimposed on the initial screen position to obtain the current screen position. Since the current screen position is obtained by mapping the user's active control displacement, it can accurately reflect the corresponding position of the user's current air gesture in the display interface.

[0039] For example, when entering the air control mode, the initial interaction position corresponding to the center point of the palm is (100 mm, 80 mm). After the user actively moves the hand, the current active control displacement is determined to be: horizontal +10 mm, vertical +5 mm. Subsequently, according to the pre-established displacement mapping rules, the above active control displacement is converted into the corresponding screen displacement: horizontal +80 pixels, vertical +40 pixels.

[0040] Furthermore, the system obtains the preset response areas corresponding to each interactive interface element (such as icons, buttons, menu items, or other interactive controls) in the current display interface, and determines whether the current screen position is within a preset response area. When the current screen position enters a preset response area, the interactive interface element corresponding to that preset response area is identified as the current control object. If the current screen position does not enter any preset response area, the current control object is kept unchanged or set to empty to wait for subsequent control position updates.

[0041] Methods for constructing gesture morphology descriptors include: The relative displacement direction and magnitude of the index finger relative to the center point of the palm are determined based on the calibration control information, and the bending state, finger opening degree, angle between adjacent fingers and the direction of the index finger are calculated based on the coordinates of key points on the user's hand. In this embodiment, it should be noted that after obtaining the calibration control information, the system first determines the relative displacement direction and magnitude of the index finger relative to the center point of the palm based on the calibration control information. Since the calibration control information has eliminated the overall displacement influence caused by changes in the user's holding posture, the relative displacement therein can accurately reflect the user's active control actions. Specifically, the relative displacement vector in the calibration control information is read, the current movement direction of the index finger is determined based on the relative displacement vector, and the corresponding relative displacement magnitude is determined based on the length of the relative displacement vector. For example, when the relative displacement vector is (20, -15), it can be determined that the index finger is currently moving in the upper right direction, with a relative displacement magnitude of approximately 25 pixels.

[0042] Subsequently, the system reads the coordinates of key points on the current user's hand. These coordinates are the skeletal key points corresponding to the user's hand in the current image frame, including the previously extracted coordinates of the palm center point, finger joints, fingertips, and other key points, used to characterize the current hand's spatial pose. The structural features of the current gesture are calculated based on the positional relationships between these key points. Specifically, the bending state of each finger is determined by the connection relationships formed between the finger joints; the degree of finger spread is determined by the distance between adjacent fingers; the angle between the direction vectors of adjacent fingers is determined by the angle between them; and the direction of the index finger is determined by the direction vector formed from the base of the index finger joint to the fingertip. For example, when the system detects that the index finger is straight, the middle finger is slightly bent, the distance between the index and middle fingers is large, and the index finger is pointing towards the right side of the screen, the structural features of the gesture—index finger straight, fingers spread, and index finger pointing to the right—can be obtained respectively.

[0043] Read the interaction attributes corresponding to the currently manipulated object, and determine the target shape parameters to describe the interaction characteristics of the currently manipulated object based on the interaction attributes; combine the target shape parameters with the relative displacement direction and relative displacement amplitude of the index finger to form the gesture shape descriptor corresponding to the current interaction process.

[0044] The system reads the interaction attributes corresponding to the currently manipulated object and determines the target shape parameters used to describe the current interactive object based on these attributes. Since different manipulated objects focus on different interaction features, only shape parameters relevant to the current interactive object are retained for subsequent descriptor construction. For example, when the current manipulated object is an image browsing area, the relative displacement direction of the index finger, the relative displacement amplitude, and the degree of finger spread can be selected as target shape parameters; when the current manipulated object is a button control, the index finger bending state, the index finger pointing direction, and the relative displacement amplitude can be selected as target shape parameters.

[0045] Finally, the determined target shape parameters, the relative displacement direction of the index finger, and the relative displacement amplitude of the index finger are combined and encoded in a preset order to form a gesture shape descriptor corresponding to the current interaction process, so as to uniformly represent the user's gesture shape and active control state under the current control object. For example, when the current control object is an image browsing area, the target shape parameters are "index finger straight, fingers spread", the corresponding relative displacement direction of the index finger is "upper right", and the relative displacement amplitude is "level three", which are then combined to form the corresponding gesture shape descriptor.

[0046] Methods for generating control commands include: Continuously acquire gesture shape descriptors corresponding to a series of consecutive image frames, and assemble the temporal change trajectory of the gesture shape descriptors in chronological order; read the set of preset interactive gestures corresponding to the current manipulated object, and establish the correspondence between each preset interactive gesture and the corresponding standard temporal change trajectory; It should be noted that the system continuously acquires gesture descriptors corresponding to several consecutive image frames and arranges them according to the acquisition time sequence of the image frames to form a temporal change trajectory of the gesture descriptors. Since a user's completion of a gesture interaction usually involves multiple stages, including the start, continuous change, and end of the gesture, using a temporal change trajectory can completely record the entire interaction process, rather than judging based on the gesture state at only a single moment. For example, if the gesture descriptors corresponding to four consecutive image frames are "index finger extended, small displacement," "index finger extended, moving to the right," "index finger extended, increased displacement," and "index finger stops moving," then these four gesture descriptors, arranged in chronological order, form the temporal change trajectory corresponding to the current interaction process.

[0047] The system reads the set of preset interactive gestures corresponding to the currently manipulated object and establishes a correspondence between each preset interactive gesture and its corresponding standard time-series change trajectory. Each preset interactive gesture corresponds to a pre-defined standard time-series change trajectory, which describes the change process of the gesture's form at each stage when the gesture is executed. Different manipulated objects correspond to different sets of preset interactive gestures. For example, when the current manipulated object is an image browsing area, its preset interactive gesture set includes gestures such as swiping left, swiping right, zooming in, and zooming out, each corresponding to its respective standard time-series change trajectory; when the current manipulated object is a video playback button, its preset interactive gesture set includes gestures such as clicking to play and pause.

[0048] The temporal change trajectory of the gesture shape descriptor is matched with each standard temporal change trajectory. Based on the matching results, interactive gestures that do not correspond to the interaction attributes of the current controlled object are filtered out. Context disambiguation is performed on the remaining interactive gestures to determine the target interactive gesture corresponding to the current interaction process. It should be noted that the system matches the temporal change trajectory of the gesture morphology descriptor corresponding to the current interaction process with various standard temporal change trajectories. Based on the matching results, interaction gestures that do not correspond to the interaction attributes of the current controlled object are filtered out. Then, context disambiguation processing is performed on the remaining interaction gestures to determine the target interaction gesture. Context disambiguation processing involves combining the current controlled object and its interaction attributes to further discriminate multiple matching results. When multiple interaction gestures have similar gesture change trajectories, the interaction gesture that matches the current controlled object's interaction method is prioritized as the target interaction gesture.

[0049] For example, if the current time-series change trajectory has a high degree of matching with both "swipe right" and "page turning", when the current object of manipulation is an image browsing area, the target interaction gesture is determined to be "swipe right" based on the interaction attributes of the image browsing area; when the current object of manipulation is an electronic document reading area, the target interaction gesture is determined to be "page turning", thereby avoiding misrecognition of the same gesture in different interaction scenarios.

[0050] Read the pre-established instruction mapping relationship between the target interactive gesture and the currently manipulated object, and generate the corresponding control instructions based on the instruction mapping relationship.

[0051] Specifically, a mapping relationship between the target interaction gesture and the system execution command is pre-established for each type of control object. Once the target interaction gesture is determined, the system looks up the corresponding mapping relationship and generates the corresponding control command, which is then sent to the tablet for execution. For example, when the current control object is an image browsing area and the target interaction gesture is "swipe right", the control command "switch to the next image" is generated based on the pre-established command mapping relationship; when the current control object is a video playback button and the target interaction gesture is "click", the control command "play / pause video" is generated, thus completing the corresponding air interaction operation.

[0052] The methods for setting up a preset gesture integrity judgment window include: Based on the standard temporal change trajectory corresponding to the target interactive gesture, the system determines the starting form, ending form, and duration range of the target interactive gesture. In this embodiment, it should be noted that the system determines the starting form, ending form, and duration range of the target interactive gesture based on the standard temporal change trajectory. The standard temporal change trajectory records the order of change of gesture form descriptors at each stage when the target interactive gesture is executed. Therefore, the starting gesture form of the standard temporal change trajectory can be taken as the starting form of the target interactive gesture, and the last gesture form can be taken as the ending form. The system also calculates the historical duration corresponding to the execution of the target interactive gesture to determine the duration range for successfully completing the interactive gesture.

[0053] For example, when the target interactive gesture is a swipe to the right, its standard timing change trajectory can be expressed as "index finger straightens and begins to move", "index finger continues to move to the right", and "index finger stops moving". The first stage is determined as the starting form and the last stage is determined as the ending form. Based on historical statistical results, the duration of the interactive gesture is determined to be within the range of 200ms to 500ms.

[0054] Based on the allowable fluctuation range of the gesture shape descriptors corresponding to each stage of the target interactive gesture, the system determines the shape tolerance range corresponding to each stage. Since users may experience slight jitter during the execution of air gestures, the actual gesture shape descriptor is not required to be completely identical to the standard gesture shape descriptor. Instead, an allowable deviation range is set for each stage. As long as the current gesture shape descriptor remains within the allowable fluctuation range of the corresponding stage, the current stage is considered to still meet the requirements of the target interactive gesture.

[0055] For example, for a right swipe gesture, during the continuous movement phase, a small deviation in the direction the index finger is pointing is allowed, and a certain fluctuation in the relative displacement amplitude is allowed. As long as it remains within the corresponding allowable fluctuation range, the current gesture shape is determined to be within the shape tolerance range corresponding to that phase.

[0056] By associating the duration range with the corresponding morphological tolerance range for each stage, a preset gesture integrity judgment window is constructed for the corresponding target interactive gesture.

[0057] Specifically, the effective execution time of the target interactive gesture is limited by a duration range, and the form requirements that the target interactive gesture should meet in different execution stages are limited by the form tolerance range of each stage. When both time constraints and form constraints are met simultaneously, a preset gesture integrity judgment window is formed for the corresponding target interactive gesture. For example, for a right swipe gesture in an image browsing area, a preset gesture integrity judgment window can be established with a duration of 200ms to 500ms, and each of the start, movement, and end stages corresponding to its own form tolerance range. Subsequently, if the user continuously meets the form tolerance requirements of each stage within this duration range, the right swipe gesture is considered to have been fully executed; if the user exits the corresponding form tolerance range prematurely, the current interaction process is considered to have ended.

[0058] Methods for continuously monitoring whether gesture shape descriptors remain within the tolerance range of the target interaction gesture include: Within the preset gesture integrity determination window, continuously acquire the gesture shape descriptor corresponding to the user's hand image in the current frame, and read the shape tolerance range of the target interaction gesture corresponding to the time sequence stage according to the current gesture shape descriptor. The current gesture shape descriptor is compared with the shape tolerance range of the corresponding stage, and the duration of the current target interaction gesture is simultaneously counted to determine whether the current gesture shape descriptor is continuously maintained within the shape tolerance range of the corresponding stage and the duration is within the duration range. When the current gesture descriptor remains within the shape tolerance range of the corresponding stage and the duration is within the duration range, the target interaction gesture is determined to be fully executed, and the corresponding interaction operation is executed according to the control command. When the current gesture descriptor deviates from the shape tolerance range of the corresponding stage, or the duration exceeds the duration range, the current interaction process is determined to end, the air control mode is turned off, and the gesture descriptor, target interaction gesture and control command corresponding to the current interaction process are cleared.

[0059] For example, if the current target of manipulation is an image browsing area, and the system has recognized the target interaction gesture as "swipe right," it will generate the control command "switch to the next image." The corresponding preset gesture integrity judgment window includes a duration range of 200ms to 500ms, as well as the shape tolerance ranges corresponding to the start phase, movement phase, and end phase.

[0060] The system continuously acquires the gesture descriptor corresponding to the current image frame within the judgment window. When it detects that the user keeps their index finger straight for approximately 320ms, with the relative displacement direction of the index finger continuously to the right, and the gesture shape at each stage is within the corresponding shape tolerance range, the system determines that the rightward swipe gesture has been fully executed and switches the image to the next one according to the control command. If the user retracts their index finger prematurely during the swipe, causing the current gesture descriptor to deviate from the shape tolerance range corresponding to the movement stage, or if the gesture is maintained for more than 500ms, the system determines that the current interaction process has ended, closes the air control mode, does not perform the image switching operation, and waits for the user to initiate the next air control.

[0061] Example 2 Please see Figure 2 As shown, parts not described in detail in this embodiment are described in Embodiment 1. A gesture recognition-based air control system for a tablet computer is provided, comprising: The intent confirmation module is used to obtain the continuous interaction state of the user's hand in the interaction space in front of the tablet computer, determine the intent of the continuous interaction state according to the preset control intent confirmation rules, and establish corresponding control benchmark reference information after confirming entry into the air control mode. The behavior calibration module calibrates the control behavior in continuous interactive states based on the control reference information, separates the influence of user active control and changes in holding posture, and obtains calibrated control information that characterizes the user's active control behavior; based on the calibrated control information, it establishes a spatial mapping relationship with the tablet computer display interface to determine the current control object. The semantic parsing module combines calibration control information and the current control object to construct a gesture shape descriptor. Based on the temporal change trajectory of the gesture shape descriptor and the correspondence between the preset interactive gestures, it performs context disambiguation to determine the target interactive gesture corresponding to the current interaction process and generate control instructions. The execution control module continuously monitors whether the gesture shape descriptor remains within the tolerance range of the target interactive gesture within the preset gesture integrity judgment window. If it remains within the tolerance range, the interactive operation is executed according to the control command. If it deviates, the current interactive process is determined to end and the air control mode is turned off.

[0062] Example 3 This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for controlling a tablet computer with gesture recognition.

[0063] Since the electronic device described in this embodiment is the one used to implement the gesture recognition-based air control method and system for tablet computers in this application, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the gesture recognition-based air control method and system for tablet computers described in this application. Therefore, how the electronic device implements the method in this application will not be described in detail here. Any electronic device used by those skilled in the art to implement the gesture recognition-based air control method and system for tablet computers in this application falls within the scope of protection of this application.

[0064] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0065] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for users of ordinary technical skills, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for air-controlling a tablet computer based on gesture recognition, characterized in that, include: S1. Obtain the continuous interaction state of the user's hand in the interaction space in front of the tablet computer, determine the intention of the continuous interaction state according to the preset control intention confirmation rule, and establish the corresponding control reference information after confirming that the air control mode has been entered. S2. Based on the control reference information, calibrate the control behavior in the continuous interaction state, separate the influence of user active control and changes in holding posture, and obtain calibration control information that characterizes the user's active control behavior; establish the spatial mapping relationship between the interaction space and the tablet computer display interface, and determine the current control object based on the calibration control information. S3. Combine calibration control information and the current control object to construct a gesture shape descriptor. Based on the correspondence between the temporal change trajectory of the gesture shape descriptor and the preset interactive gesture, perform context disambiguation processing to determine the target interactive gesture corresponding to the current interaction process and generate control instructions. S4. Within the preset gesture integrity determination window, continuously monitor whether the gesture shape descriptor remains within the tolerance range of the target interactive gesture. If it remains within the tolerance range, execute the interactive operation according to the control command. If it deviates, determine that the current interactive process has ended and close the air control mode.

2. The method for air-controlling a tablet computer based on gesture recognition according to claim 1, characterized in that, The method for obtaining the continuous interaction state includes: The tablet's front-facing camera continuously captures images of the user's hand in the interactive space in front of the display screen. The hand in each frame is tracked and identified, and the corresponding key point coordinates are extracted. A sequence of hand movement changes is established based on the key point coordinates of each consecutive frame, which serves as a continuous interactive state representing the user's hand position and gesture changes.

3. The method for air-controlling a tablet computer based on gesture recognition according to claim 2, characterized in that, The method for obtaining the control reference information includes: The overall hand motion velocity vector is calculated based on the continuous interaction state, and the intention is determined by combining the preset control intention confirmation rules; the preset control intention confirmation rules include a preset static intention threshold and a preset intention confirmation frame number. The magnitude of the overall hand movement velocity vector is continuously compared with a preset static intention threshold. When the magnitude is continuously lower than the preset static intention threshold and the preset intention confirmation frame number is reached, it is determined that the user has the intention to enter the air control mode. The coordinates of the palm center point and the index finger tip corresponding to the user's hand image in the current confirmation frame are read. A control reference point pair is constructed based on the palm center coordinates and the index finger tip coordinates as the control reference information to activate the air control mode.

4. The method for air-controlling a tablet computer based on gesture recognition according to claim 3, characterized in that, The method for obtaining calibration control information characterizing user active control behavior includes: The coordinates of the current palm center point and the current index fingertip are obtained based on the continuous interaction state, and the overall displacement vector and the comprehensive displacement vector are calculated relative to the control reference information. The comprehensive displacement vector is calibrated based on the overall displacement vector to eliminate the influence of the overall displacement caused by the change of holding posture, and the relative displacement vector of the index fingertip coordinates relative to the palm center point is obtained. The relative displacement vector is used as the calibration control information.

5. A method for air-controlling a tablet computer based on gesture recognition according to claim 4, characterized in that, The method for determining the current manipulated object includes: Based on the control reference information, determine the initial hand interaction posture when entering the air control mode, and establish the correspondence between the initial hand interaction posture and the preset initial screen position to form the spatial mapping starting point between the interaction space and the tablet computer display interface; establish the displacement mapping rules between the interaction space displacement and the screen displacement based on the spatial mapping starting point. Based on the calibration control information, determine the user's active control displacement relative to the initial interaction posture of the hand, and obtain the corresponding current screen position according to the displacement mapping rule; obtain the preset response area corresponding to each interactive interface element in the current display interface, and determine the interactive interface element corresponding to the preset response area into which the current screen position falls as the current control object.

6. A method for air-controlling a tablet computer based on gesture recognition according to claim 5, characterized in that, The method for constructing the gesture morphology descriptor includes: The relative displacement direction and magnitude of the index finger relative to the center point of the palm are determined based on the calibration control information, and the bending state, finger opening degree, angle between adjacent fingers and the direction of the index finger are calculated based on the coordinates of key points on the user's hand. Read the interaction attributes corresponding to the currently manipulated object, and determine the target shape parameters to describe the interaction characteristics of the currently manipulated object based on the interaction attributes; combine the target shape parameters with the relative displacement direction and relative displacement amplitude of the index finger to form the gesture shape descriptor corresponding to the current interaction process.

7. A method for controlling a tablet computer with air gestures based on claim 6, characterized in that, The method for generating control commands includes: Continuously acquire gesture shape descriptors corresponding to a series of consecutive image frames, and assemble the temporal change trajectory of the gesture shape descriptors in chronological order; read the set of preset interactive gestures corresponding to the current manipulated object, and establish the correspondence between each preset interactive gesture and the corresponding standard temporal change trajectory; The temporal change trajectory of the gesture shape descriptor is matched with each standard temporal change trajectory. Based on the matching results, interactive gestures that do not correspond to the interaction attributes of the current controlled object are filtered out. Context disambiguation is performed on the remaining interactive gestures to determine the target interactive gesture corresponding to the current interaction process. Read the pre-established instruction mapping relationship between the target interactive gesture and the currently manipulated object, and generate the corresponding control instructions based on the instruction mapping relationship.

8. A method for controlling a tablet computer with air gestures based on claim 7, characterized in that, The method for setting the preset gesture integrity determination window includes: Based on the standard temporal change trajectory corresponding to the target interactive gesture, determine the starting form, ending form, and duration range of the target interactive gesture; based on the allowable fluctuation range of the gesture form descriptor corresponding to each stage of the target interactive gesture, determine the form tolerance range corresponding to each stage; associate the duration range with the form tolerance range corresponding to each stage to construct a preset gesture integrity judgment window for the target interactive gesture.

9. A method for controlling a tablet computer with air gestures based on claim 8, characterized in that, The method for continuously monitoring whether the gesture shape descriptor remains within the tolerance range of the target interactive gesture includes: Within the preset gesture integrity determination window, continuously acquire the gesture shape descriptor corresponding to the user's hand image in the current frame, and read the shape tolerance range of the target interaction gesture corresponding to the time sequence stage according to the current gesture shape descriptor. The current gesture shape descriptor is compared with the shape tolerance range of the corresponding stage, and the duration of the current target interaction gesture is simultaneously counted to determine whether the current gesture shape descriptor is continuously maintained within the shape tolerance range of the corresponding stage and the duration is within the duration range. When the current gesture descriptor remains within the shape tolerance range of the corresponding stage and the duration is within the duration range, the target interaction gesture is determined to be fully executed, and the corresponding interaction operation is executed according to the control command. When the current gesture descriptor deviates from the shape tolerance range of the corresponding stage, or the duration exceeds the duration range, the current interaction process is determined to end, the air control mode is turned off, and the gesture descriptor, target interaction gesture and control command corresponding to the current interaction process are cleared.

10. A gesture recognition-based air control system for a tablet computer, implemented by any one of claims 1 to 9, characterized in that, include: The intent confirmation module is used to obtain the continuous interaction state of the user's hand in the interaction space in front of the tablet computer, determine the intent of the continuous interaction state according to the preset control intent confirmation rules, and establish corresponding control benchmark reference information after confirming entry into the air control mode. The behavior calibration module calibrates the control behavior in continuous interactive states based on the control reference information, separates the influence of user active control and changes in holding posture, and obtains calibrated control information that characterizes the user's active control behavior; based on the calibrated control information, it establishes a spatial mapping relationship with the tablet computer display interface to determine the current control object. The semantic parsing module combines calibration control information and the current control object to construct a gesture shape descriptor. Based on the temporal change trajectory of the gesture shape descriptor and the correspondence between the preset interactive gestures, it performs context disambiguation to determine the target interactive gesture corresponding to the current interaction process and generate control instructions. The execution control module continuously monitors whether the gesture shape descriptor remains within the tolerance range of the target interactive gesture within the preset gesture integrity judgment window. If it remains within the tolerance range, the interactive operation is executed according to the control command. If it deviates, the current interactive process is determined to end and the air control mode is turned off.

Images