Glasses volley interaction method and system integrated with gesture recognition module

Abstract

The invention discloses a glasses volley interaction method and system integrated with a gesture recognition module, and the method comprises the steps: obtaining space calibration, integrating a TOF laser ranging module at the front end of a wearable glasses frame, and enabling the TOF laser ranging module to be electrically connected with a main control unit of the glasses; the TOF laser ranging module scans a field of view in front of the glasses and collects reference depth data, establishes a mapping relation between a TOF measurement space and a target display screen through a calibration program, and stores the mapping relation as a reference model for subsequent position projection; the TOF laser ranging module continuously collects depth frame data according to a preset sampling frequency, the main control unit compares a current depth frame with the reference depth data and extracts a foreground area according to a preset depth difference threshold value, a user does not need to wear additional equipment such as a ring, a bracelet or a glove, and the user experience is improved. Gesture recognition and screen interaction can be completed only by means of the TOF laser ranging module arranged in the glasses.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human-computer interaction and smart wearable device technology, in particular to a glasses air interaction method integrated with a gesture recognition module and a system thereof. BACKGROUND

[0002] The common ways of interacting with a display screen at present include: Touch interaction: direct contact with the screen is required, which is limited by the use environment; Mouse and keyboard control: cumbersome and inconvenient to operate; Smart ring / handband interaction: requires wearing an additional device, and the operation is indirect; Vision-based air interaction: no medium is required, but the recognition accuracy and stability are insufficient.

[0003] Especially in the field of mobile office and augmented reality (AR), users hope to interact with the screen through natural gestures without any additional control medium. However, the existing air interaction technology has obvious deficiencies in spatial positioning accuracy, recognition delay, light adaptability, and motion mapping consistency. Therefore, a glasses air interaction method integrated with a gesture recognition module and a system thereof are proposed. SUMMARY

[0004] The purpose of the present application is to provide a glasses air interaction method integrated with a gesture recognition module and a system thereof. By embedding a TOF laser ranging module in the glasses frame, the space in front of the eyes is scanned in depth, the finger motion is recognized and mapped to the screen cursor in real time, and natural human-computer interaction is realized without wearing a ring, handband or gloves.

[0005] To achieve the above purpose, the first aspect of the embodiments of the present disclosure is to solve the above technical problems; One technical solution adopted by the present application is: a glasses air interaction method integrated with a gesture recognition module, comprising the following steps: Obtaining spatial calibration, a TOF laser ranging module is integrated in the front end of the wearable glasses frame, the TOF laser ranging module is electrically connected with the master control unit of the glasses; the TOF laser ranging module scans the field of view in front of the glasses and collects reference depth data, at the same time, a mapping relationship between the TOF measurement space and the target display screen is established through a calibration program, and the mapping relationship is stored as a reference model for subsequent position projection; Based on the TOF laser ranging module, depth frame data is continuously collected at a predetermined sampling frequency, the master control unit compares the current depth frame with the reference depth data and extracts the foreground area according to the pre-set depth difference threshold, the foreground area is used as a candidate area for the presence of hands or fingers; The foreground region is spatially clustered and analyzed for connected domain, three-dimensional key points are calculated and temporally tracked between consecutive frames, and a filtering-based smoothing process is used to suppress ranging noise and output stable three-dimensional position trajectories; Based on the three-dimensional position trajectories, trajectory features are extracted according to predetermined time and space windows, and the trajectories are mapped to predefined gesture categories according to preset rules or trained classifiers, and the current gesture is determined to be up, down, left, right, click, and slide; According to the gesture category and the position of the finger in the mapping relationship, a command package corresponding to the display screen interaction is generated and sent to the display control unit through the main control unit of the glasses in a bus or wireless manner, and the display control unit performs corresponding cursor movement, page scrolling, click confirmation and sliding switching interaction on the display device according to the received command, and simultaneously feeds back the execution result to the user in a visual or tactile manner.

[0006] In a possible implementation manner, the mapping relationship is obtained by collecting a set of calibration samples at different viewing distances and angles in the initialization stage and fitting by using a polynomial regression or a bilinear interpolation method, to compensate for the nonlinear error and optical distortion of the TOF laser ranging module, thereby ensuring stable mapping relationship from finger spatial coordinates to display screen coordinates.

[0007] In a possible implementation manner, the depth difference threshold is an adaptive threshold, which is adjusted in real time based on ambient light intensity, TOF reflectivity reading and historical baseline noise level, to improve the robustness of foreground extraction in strong light, shadow or high-reflectivity background conditions.

[0008] In a possible implementation manner, the filtering process uses weighted Kalman filtering or exponential smoothing algorithm, and the process noise and measurement noise covariance parameters of the filter are dynamically updated according to the current sampling stability through an adaptive strategy, to balance tracking smoothness and dynamic response speed; the three-dimensional key points include hand centroid and fingertip key points, and in a fine finger interaction scenario, the fingertip key points are preferentially used for high-precision position mapping, to improve the recognition accuracy of click and micro-motion actions.

[0009] In a possible implementation manner, the interaction action determination adopts a hybrid determination strategy: in a simple scenario, a threshold-based rule determination is preferentially used to reduce computational resource consumption, and in a complex or variable scenario, a trained temporal classifier (such as a model based on LSTM or HMM) is preferentially used to improve the recognition rate of complex gestures and continuous actions.

[0010] In a possible implementation, the trajectory features include displacement, speed, direction, and depth change features, and the classifier has an online incremental learning capability, when the system detects repeated correction or misrecognition of a gesture by a user, the latest trajectory features and label information of the gesture are temporarily stored locally and used for subsequent model fine-tuning or uploaded to the cloud to participate in retraining of a global model, thereby realizing personalized adaptation.

[0011] In a possible implementation, the command packet includes a unique instruction identifier, a gesture type identifier, screen coordinates of a current finger mapping, and a timestamp, and is transmitted to a display control unit in a data frame format containing a check bit on a communication link such as I2C, SPI, UART, or Bluetooth Low Energy, and the display control unit performs a corresponding interactive action according to a current user interface context after receiving the command.

[0012] In a possible implementation, the method further includes: when the display control unit performs an interactive action, evaluating the correctness of the last action recognition based on subsequent actions or explicit feedback information (such as eye movement confirmation, voice confirmation, or gesture cancellation) of the user, and using the evaluation result for local parameter adjustment or submitting to the cloud for offline model optimization.

[0013] In a possible implementation, the TOF laser ranging module is in a low-power scanning or sleep mode when no gesture is detected, and enters a high-frequency sampling mode when a depth change exceeds a preset wake-up threshold, thereby realizing a low-power working mechanism and prolonging the endurance time of the wearable glasses device.

[0014] In a possible implementation, the method supports multi-point and multi-finger recognition, can track and distinguish independent trajectories of two or more fingers in parallel, and further recognize composite gestures such as double-finger zooming, two-finger rotating, or double-finger dragging and map them to corresponding zooming, rotating, or dragging interactive commands.

[0015] In a possible implementation, near-infrared structured light or active infrared illumination is used to compensate for TOF ranging errors in strong light or high-reflectivity environments; monocular or binocular RGB cameras are combined to realize visual-depth joint recognition in scenes where recognition accuracy needs to be improved; and in a long-distance interaction scenario, a larger range of gesture recognition coverage is realized by expanding the TOF field of view or combining multi-module collaborative perception.

[0016] A second aspect of the embodiments of the present disclosure is to solve the above technical problems. Another technical solution adopted by the present application is: an eyeglass air interaction system integrated with a gesture recognition module, comprising: The calibration mapping module is configured to acquire a space calibration, and a TOF laser ranging module is integrated at a front end of a wearable glasses frame, and the TOF laser ranging module is electrically connected with a master control unit of the glasses; the TOF laser ranging module scans a field of view in front of the glasses and collects reference depth data, and simultaneously establishes a mapping relationship between a TOF measurement space and a target display screen through a calibration program, and stores the mapping relationship as a reference model for subsequent position projection; The frame difference extraction module is configured to continuously collect depth frame data at a predetermined sampling frequency based on the TOF laser ranging module, and the master control unit compares a current depth frame with the reference depth data and extracts a foreground region according to a preset depth difference threshold, and the foreground region is used as a candidate region where a hand or a finger exists; The trajectory clustering and smoothing module is configured to perform spatial clustering and connected component analysis on the foreground region, calculate three-dimensional key points, perform time sequence tracking between continuous frames, and simultaneously adopt filtering-based smoothing processing to suppress ranging noise and output a stable three-dimensional position trajectory; The gesture recognition module is configured to extract trajectory features according to a predetermined time window and a spatial window based on the three-dimensional position trajectory, and map the trajectory to a predefined gesture category according to a preset rule or a trained classifier, and determine that a current gesture is up, down, left, right, click or slide; The command feedback module is configured to generate a command package corresponding to display screen interaction according to the gesture category and a position of a finger in the mapping relationship, and send the command package to a display control unit through the master control unit of the glasses in a bus or wireless manner, and the display control unit performs corresponding cursor moving, page scrolling, click confirmation and sliding switching interaction actions on a display device according to the received command, and simultaneously feeds back execution results to a user in a visual or tactile manner.

[0017] A third aspect of the embodiments of the present disclosure is that: to solve the above technical problems, another technical solution adopted by the present application is: an electronic device, comprising a processor, a memory and a communication interface, the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned any one integrated gesture recognition module glasses hovering interaction method are implemented.

[0018] A fourth aspect of the embodiments of the present disclosure is that: to solve the above technical problems, another technical solution adopted by the present application is: a computer readable storage medium, the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned any one integrated gesture recognition module glasses hovering interaction method are implemented.

[0019] Compared with the prior art, the present application has the following advantages: The user of the present application does not need to wear additional equipment such as a ring, a bracelet or a glove, and can complete gesture recognition and screen interaction only by relying on the TOF laser ranging module built in the glasses itself. This design eliminates the limitation of traditional external equipment, and makes the interaction more convenient and free, and is especially suitable for environments that require lightness and high portability. The present application adopts TOF depth modeling and Kalman filtering, and the motion positioning accuracy reaches ±2mm. Through accurate depth data and continuous Kalman filtering, the error caused by light changes or gesture shaking in other recognition methods can be effectively avoided, so that the gesture operation is more accurate and reliable.

[0020] The present application realizes one-to-one mapping between the movement of fingers or hands and the screen cursor, and the gesture made by the user in the air can be directly reflected in the cursor or interface operation of the display screen. This seen-is-controlled interaction mode improves the user experience, and the user can operate more intuitively and naturally without physical touch or intermediate equipment, which meets the natural interaction habit of human beings.

[0021] The present application has strong environmental adaptability and can stably recognize gestures under different light conditions. In weak light or strong light environment, the system can automatically adjust the threshold and depth measurement parameters to ensure the stability and accuracy of gesture recognition, greatly improving the applicability of the system in various complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 It is a flowchart of the air interaction method of the glasses integrated with the gesture recognition module of the present application. Figure 2 It is a module diagram of the air interaction system of the glasses integrated with the gesture recognition module of the embodiment two of the present application. Figure 3 It is a structural schematic diagram of an electronic device. Figure 4 It is a structural schematic diagram of the glasses frame front end integrated with the TOF laser ranging module of the present application. Figure 5 It is a circuit diagram of the TOF laser ranging module of the present application. DETAILED DESCRIPTION

[0024] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0025] Figure 1 is a flowchart of a method for air interaction of glasses integrating a gesture recognition module according to an embodiment of the present application. It should be noted that the method of the present application is not limited to the order of the flowchart shown Figure 1 .

[0026] Embodiment one: As Figure 1 , Figure 4 and Figure 5 is a method for air interaction of glasses integrating a gesture recognition module according to the present application, comprising the following steps: In step S10, the space calibration is acquired, a TOF laser ranging module is integrated at the front end of the frame of the wearable glasses, the TOF laser ranging module is electrically connected with the master control unit of the glasses; the TOF laser ranging module scans the field of view in front of the glasses and collects the reference depth data, at the same time, a mapping relationship between the TOF measurement space and the target display screen is established through a calibration program, and the mapping relationship is stored as a reference model for subsequent position projection; In the embodiment of the present disclosure, the TOF laser ranging module is embedded at the front end of the glasses frame; the module field of view angle is , and the scanning range is 0.1-1.0m; the module is connected with the master control board of the glasses through the I2C bus; the master control chip is a low-power MCU or So C (such as STM32 / ESP32).

[0027] In the embodiment of the present disclosure, when the user wears the glasses, the space calibration is started, and the space mapping relationship between the display screen and the TOF irradiation area is established.

[0028] For each pixel point (u, v), the TOF depth data is converted into the world coordinates by the following formula: ; Wherein: : camera intrinsic matrix; : depth value measured by TOF; : position of the finger in the space coordinate system.

[0029] In the embodiment of the present disclosure, the cursor mapping model is established: ; Wherein: : screen cursor coordinates; : nonlinear mapping function obtained by calibration, fitted by polynomial regression.

[0030] For example, when the user wears the glasses and turns on the system, the TOF laser ranging module starts working. The TOF module embedded in the glasses emits infrared laser pulses to illuminate the space in front of the glasses and calculates the distance data (depth map) from each pixel point to the module according to the reflection time of the laser pulses.

[0031] The TOF module collects depth data at a certain sampling frequency (e.g., 30 Hz), and the collected data is the depth information of each pixel point, representing the distance from the TOF module to the object surface. For example, assuming a depth value of 200 mm, it means that the distance from this point to the module is 200 mm.

[0032] The reference depth map is calculated using data collected from multiple perspectives. The reference depth map is obtained by repeatedly collecting data in a fixed area and averaging the depth values to eliminate the influence of environmental changes (such as light differences and background clutter) on depth measurement.

[0033] For example: Assuming that during the calibration phase, the reference depth data of the area in front of the glasses frame is : For pixel coordinates , the reference depth value is ; For pixel coordinates , the reference depth value is .

[0034] During the initialization of the calibration process, each depth data point obtained by the TOF module needs to be mapped to the pixel coordinates on the display screen.

[0035] The goal of this process is to ensure that the movement of the finger or hand in the space in front of the glasses can be accurately aligned with the movement of the screen cursor. To this end, a spatial geometric transformation algorithm is used to convert the depth data measured by the TOF module into the coordinate system of the display screen.

[0036] For example, assuming that the resolution of the display screen is , and the measurement range of the TOF module is 0.1-1.0 m, according to the distance and field of view angle, each depth point corresponds to the screen coordinates . This mapping function can be automatically obtained by a calibration algorithm, and usually uses polynomial regression or bilinear interpolation for optimization.

[0037] After calibration, the mapping relationship Will be stored in the host unit.

[0038] The mapping model is used for real-time processing of each subsequent frame of depth data, ensuring that each captured gesture action is accurately reflected on the screen cursor position.

[0039] For example: assume in the calibration process, the TOF module calculates a point in space according to the depth data The mapping relationship of the point is: Depth 300 mm from the screen); The mapped screen cursor position is .

[0040] When the finger is 300 mm from the glasses, the cursor position on the screen should appear at coordinates (100, 200).

[0041] When the user performs a gesture operation, the TOF module will obtain the position of the finger in space in real time, and through the established mapping model, the position will be converted to cursor movement on the screen, thus completing the interaction between the gesture and the display screen.

[0042] Once the calibration process is complete, it will enter normal working mode.

[0043] In this mode, each gesture action will go through depth recognition, feature extraction and classification by the TOF module, and will be combined with the previously established spatial mapping model to achieve real-time mapping of the finger or hand position in space and screen interaction.

[0044] For example: when the user waves the finger upwards, according to the spatial position of the finger (depth map data provided by the TOF module), the finger movement is converted to an upward movement of the cursor on the screen through the mapping model, achieving the desired interaction effect.

[0045] In the establishment of spatial calibration and mapping relationship.

[0046] Through the high-precision depth measurement of the TOF module and the multi-view calibration algorithm, the high consistency between each user gesture action and the cursor position on the screen is ensured.

[0047] This process solves the problem of spatial alignment between gestures and the screen, and provides accurate basic data support for subsequent interaction operations (such as clicking, sliding, scrolling, etc.).

[0048] Through calibration, high-precision interaction performance can be maintained under different environments and different users, while avoiding errors or deviations caused by changes in external conditions or differences in user behavior.

[0049] In step S20, the TOF laser ranging module continuously collects depth frame data at a predetermined sampling frequency. The master control unit compares the current depth frame with the reference depth data and extracts the foreground region according to a pre-set depth difference threshold. The foreground region is a candidate region where a hand or a finger exists. In the embodiments of the present disclosure, the current frame is calculated by the background difference: ; Wherein: : depth change threshold (default 20mm); : binary mask.

[0050] In the embodiments of the present disclosure, the centroid calculation and filtering: ; The weighted Kalman filter is used to denoise the hand centroid coordinates: ; Wherein: : Kalman gain; : system noise covariance; For example, at startup, the TOF (Time-of-Flight) laser ranging module starts working and continuously acquires depth data in front of the glasses.

[0051] Assume that the reference depth data of the area in front of the glasses is : For pixel coordinates , the reference depth value is ; For pixel coordinates , the reference depth value is .

[0052] These reference depth values represent the depth measurement values of the background or fixed objects when there is no hand or finger present.

[0053] During operation, the TOF module continuously acquires depth data in front of the glasses at a certain frequency (such as 30Hz), forming continuous depth frame data .

[0054] For example, the depth values of the current depth frame may be as follows: For pixel coordinates , the current depth value is ; For pixel coordinates , the current depth value is .

[0055] The depth data of the current frame is compared with the reference depth data depth data is calculated for each pixel: ; For example, for pixel point , the depth difference is: ; For pixel point , the depth difference is: ; To extract the foreground region of the hand or finger, a depth difference threshold is set (hypothetically 20mm). When the depth difference of a certain pixel point exceeds this threshold, it is considered that the region is foreground, i.e. the region of the hand or finger. At this time, the region will be marked as a foreground region.

[0056] For example, if the set threshold is , then: For pixel point , because , the depth difference is exactly equal to the threshold, it is considered that this point may belong to the foreground; For pixel point , because , the depth difference is also , so this point is also considered to be foreground.

[0057] If the depth difference of a certain point exceeds the set threshold , the pixel is marked as a foreground region. If it does not exceed the threshold, it is considered to be a background region.

[0058] Through depth difference calculation and threshold judgment, all pixel points that meet the conditions are marked as foreground regions. These foreground pixel points represent the position of the hand or finger.

[0059] For example, after threshold screening, the marked foreground region is as follows: For pixel , the depth difference meets the threshold condition and is marked as foreground; For pixel , the depth difference meets the threshold condition and is marked as foreground.

[0060] These marked foreground regions will be used as candidate regions of the hand or finger for subsequent action recognition and trajectory analysis.

[0061] After extracting the foreground region, further morphological processing (such as erosion, dilation) is performed on the foreground region to remove noise, ensuring that the extracted foreground region is a continuous gesture trajectory.

[0062] For example, assume the user places a finger in front of the glasses at a certain position and makes a right swipe motion.

[0063] In this process, the TOF module captures depth data of the finger motion in real time, and as the finger moves, the depth difference changes continuously.

[0064] When the finger approaches the glasses, the depth value changes significantly, causing the depth difference to exceed the set threshold .

[0065] For example, the depth of the finger changes from the original 300mm (background) to 350mm (foreground of the finger).

[0066] At this point, the system marks the area where the finger is as foreground and tracks the movement trajectory of the finger in subsequent steps, ultimately converting it to an interactive action on the screen, such as "right swipe".

[0067] Through continuous depth frame acquisition and foreground region separation, the background and foreground regions can be effectively distinguished, and the position and movement trajectory of the hand or finger can be accurately identified.

[0068] The setting of the depth difference threshold is the core of this process, determining the extraction accuracy of the foreground and the interaction effect.

[0069] Subsequent gesture recognition and action mapping will be based on the tracking results of this foreground region, ensuring that the system can accurately respond to user gesture operations.

[0070] In step S30, the foreground region is spatially clustered and connected component analyzed, three-dimensional key points are calculated and temporally tracked between consecutive frames, and filter-based smoothing processing is used to suppress ranging noise and output stable three-dimensional position trajectories.

[0071] In the embodiments of the present disclosure, the velocity vector of the centroid in consecutive frames is calculated: ; And the direction angle is calculated: ; In the embodiments of the present disclosure, the action mode recognition parameters are: displacement amplitude ; average speed ; time window length ; If the minimum recognition threshold is exceeded , the system determines that the action is valid.

[0072] In the embodiments of the present disclosure, the feature vector is established: The feature vector is used as input for the next classifier to determine the action type: For example, the features of the hand or fingers are extracted from the foreground region, such as three-dimensional centroid, key points, etc., and time series tracking is performed for subsequent gesture recognition and interactive operation.

[0073] Objective: Extract the position features of the hand or fingers from the foreground region; Trajectory tracking is performed on the hand / fingers in consecutive frames to ensure the stability of the interactive operation; A filtering method is used to smooth the tracking trajectory and reduce noise interference.

[0074] It is assumed that step B has been completed, i.e., the foreground region has been extracted through depth difference calculation and threshold setting. The foreground region usually contains the user's hand or fingers, which is a key part of the finger (such as the fingertip) or the centroid of the entire hand.

[0075] For example, it is assumed that the user's finger is in front of the glasses, and the following foreground regions are extracted from the field of view: Foreground region 1: the finger is located , i.e., the finger is about mm away from the glasses. Foreground region 2: the finger is located , i.e., the finger moves to the right, and the depth value changes to 305 mm.

[0076] The three-dimensional position of the finger is determined through these foreground regions, and the motion of the finger is tracked in consecutive frames.

[0077] In each frame of data, the pixel points of the foreground region are extracted to form a connected region. Cluster analysis is performed on these connected regions to calculate the centroid of each region as the three-dimensional position of the hand or finger.

[0078] Each independent connected part in the foreground region is extracted using a connected region algorithm (such as the 8-neighborhood algorithm) to form a finger region.

[0079] It is assumed that the finger region is extracted as follows: Region : contains points ; Region : contains points .

[0080] The three-dimensional centroid (barycenter) of each connected domain is calculated.

[0081] For example, the centroid corresponding to region A and region B can be calculated by the weighted average formula:

[0082] wherein, is the centroid position, is the three-dimensional coordinate of each pixel point, is the number of pixel points in the foreground region.

[0083] For example, assume that region contains 3 pixel points, respectively: ; ; ; The centroid of region is calculated as follows: ; Similarly, region B will also calculate the corresponding centroid. In each frame, the three-dimensional centroid of the finger is extracted, which needs to be time-series tracked to ensure accurate tracking of finger movement in consecutive frames.

[0084] In order to avoid trajectory deviation caused by noise, jitter or depth measurement error, a filtering algorithm will be used for smoothing processing, thereby providing a more stable finger movement trajectory.

[0085] Kalman filter is used to smooth the continuous centroid position, eliminating ranging noise and errors in motion. The main idea of Kalman filter is to gradually correct the trajectory by predicting and updating each frame, thereby outputting a more stable centroid trajectory.

[0086] In each frame of data, the Kalman filter will correct the trajectory estimate value

[0087] according to the estimated position of the previous frame and the current measurement value using the filter gain : ; wherein: is the estimated position of the previous frame; is the measured finger position of the current frame; is the Kalman gain, which is dynamically adjusted according to the noise level.

[0088] After Kalman filtering, a stable three-dimensional position trajectory will be obtained, assuming that the filtered centroid trajectory is as follows: Frame 1: ; Frame 2: ; Frame 3: ; Frame 4: .

[0089] These filtered centroid positions can accurately reflect the motion trajectory of the finger in three-dimensional space.

[0090] After the feature extraction and trajectory tracking of the hand / finger are completed, the trajectory information is transmitted to the gesture recognition and mapping system of the next step to determine the gesture and generate the interactive instruction.

[0091] At this time, the motion trajectory of the finger or hand will be input signal, further mapped to the cursor position of the display screen or trigger interactive action.

[0092] Through spatial clustering, connected domain analysis, centroid calculation and Kalman filtering, the position features of the finger or hand can be extracted from the depth map obtained by the TOF module, and the features are stably time-series tracked in the continuous depth frames.

[0093] In step S40, based on the three-dimensional position trajectory, the trajectory features are extracted according to the predetermined time window and spatial window, and the trajectory is mapped to the predefined gesture category according to the preset rule or the trained classifier, and the current gesture is determined to be up, down, left, right, click and slide. In the embodiment of the present disclosure, the preset gesture set includes: up, down, left, right, click, and slide

[0094] In the embodiment of the present disclosure, the acceleration, direction and depth change are used to establish logical judgment:

[0095] Wherein: : direction speed threshold; : depth change threshold of click action; : sliding action threshold.

[0096] In the embodiment of the present disclosure, after the action recognition, the instruction package is generated: ; Sent to the main control unit through the I2C / UART bus.

[0097] For example, through the processes of depth data collection, foreground extraction, and hand centroid tracking, the continuous motion trajectories of fingers or hands have been obtained. The goal of motion recognition and classification is to extract features such as displacement, velocity, direction, and acceleration from these continuous trajectories and map them into pre-defined gesture categories, completing the recognition of gestures.

[0098] By analyzing the motion trajectories of fingers or hands, several key motion features are extracted.

[0099] These features typically include: Displacement: the change in position of fingers or hands from the current position to the next position; Velocity: the relationship between displacement change and time, representing the moving speed of fingers or hands; Direction: the direction of the movement of fingers or hands, usually represented by an angle; Acceleration: the change in velocity per unit time, reflecting the acceleration or deceleration of fingers or hands.

[0100] For example, assume that the user performs an upward gesture in front of the glasses. After the previous steps (foreground extraction, centroid calculation, smoothing filtering, etc.), the continuous centroid position sequence is obtained.

[0101] Assume the positions of the fingers in the 1st frame and the 2nd frame are as follows: 1st frame: finger position ; ( The coordinates represent the spatial position of the finger); 2nd frame: finger position .

[0102] The system calculates the displacement: ; ; This displacement represents the actual moving distance of the finger between the two frames.

[0103] Next, the velocity is calculated: ; Assuming the time difference seconds, the velocity is .

[0104] After obtaining the displacement, velocity, and other features, the gesture type is determined through pre-set rules or trained classifiers.

[0105] For example, assume that a simple classification method based on rules is used: Up: if the change of y-coordinate is greater than a certain threshold (assume 10mm) and the speed is slow (e.g. <50mm / s) between two consecutive frames, it is determined as "up"; Down: if the change of y-coordinate is negative and the speed is slow, it is determined as "down"; Down: if the change of y-coordinate is negative and the speed is slow, it is determined as "down"; Click: if the displacement is very small and the acceleration changes very fast, it is determined as "click"; Slide: if the displacement is large and the speed continues to exceed a certain threshold, it is determined as "slide".

[0106] For example: suppose the following features are detected from the last frame to the current frame: Displacement: ; Speed: ; Change of y-coordinate: (indicating that the finger moves up by 10mm); ; Speed is lower than 50mm / s.

[0107] According to these features, it is determined that the gesture is "up" and it is mapped to the corresponding operation according to the set rules, for example "page scroll up".

[0108] For more complex gestures such as "left swipe", "right swipe" or "rotation" gestures, machine learning models such as support vector machines (SVM) or deep neural networks (DNN) can be used for recognition.

[0109] For example, suppose the user performs a left swipe gesture, and the system calculates the following features based on the trajectory of the finger: Displacement: 50mm; Speed: 100mm / s; Horizontal displacement (x-direction) is greater than vertical displacement (y-direction).

[0110] Then, these features are input into the trained SVM classifier, and the SVM compares the features with the gesture labels in the historical data to predict that the gesture is "left swipe".

[0111] After identifying the gesture type through rules or classifiers, the corresponding interactive operation is triggered according to the gesture. For example: Up: scroll the webpage or menu up; Down: scroll the webpage or menu down; Left swipe: switch to the next page; Right swipe: switch to the next page; Click: click the confirmation button or option; Slide: Dragging an object on the screen.

[0112] According to the movement trajectory of the finger or hand, relevant features (such as displacement, speed, direction change, etc.) are extracted, and the gesture type is determined through a pre-set rule or classifier.

[0113] For simple gestures (such as up, down, left, right, etc.), rules are used for determination; for more complex gestures (such as sliding, rotating, etc.), machine learning algorithms can be used for classification and recognition.

[0114] In step S50, according to the gesture category and the position of the finger in the mapping relationship, a command package corresponding to the display screen interaction is generated and sent to the display control unit through the bus or wireless mode of the main control unit of the glasses, and the display control unit executes the corresponding cursor movement, page scrolling, click confirmation and sliding switching interaction on the display device according to the received command, and at the same time, the execution result is fed back to the user in a visual or tactile manner.

[0115] In the embodiment of the present disclosure, after the main control unit receives the command, the UI event trigger is performed according to the action type: Up / Down: Scroll the page; Left / Right: Switch interface; Click: Confirm; Slide: Drag the object.

[0116] In the embodiment of the present disclosure, according to the mapping function , the finger space position is projected into the screen cursor position, realizing synchronous movement.

[0117] That is: ; In the embodiment of the present disclosure, if the user feedback action misidentification, the system records the current feature and adjusts the threshold parameter, and uses the incremental KNN algorithm for local learning update.

[0118] The gesture category and finger position mapping relationship are converted into specific interaction commands, which will drive the display device (such as screen) to execute the corresponding interaction operation.

[0119] For example, if the gesture recognition is "click", the command will trigger the click event to execute the button confirmation or page jump; if the gesture is "left slide" or "right slide", the page switching operation will be executed.

[0120] The key of this step is to generate interaction commands and transmit them to the display control unit through the main control unit of the glasses, and the latter executes the interaction action according to the received command.

[0121] After gesture recognition is completed, the gesture must be mapped to a corresponding display operation.

[0122] This mapping process involves converting the recognized gesture category (e.g., move up, move down, click, swipe, etc.) into specific command packets that are transmitted to the display control unit via a bus or wireless transmission.

[0123] For example, assume the user makes a "move up" gesture, i.e., the finger moves 10 mm upwards. The system has recognized this action as a "move up" gesture through previous steps (e.g., finger trajectory extraction and motion classification) and calculated the spatial position of the finger through depth mapping.

[0124] Based on this information, an interaction command packet is generated with the following content: Command type: Move Up Position coordinates: Finger position (e.g., finger position mapped to cursor coordinates on the screen, ); Timestamp: Time of event occurrence (e.g.: 2025-11-13 10:30:00) The command packet is transmitted to the display control unit via a bus or wireless protocol.

[0125] At the receiving end, the display control unit parses the command packet and extracts information such as gesture type, cursor position, and timestamp, and then executes the corresponding interaction operation.

[0126] For example: The user makes a "move up" gesture, and the generated command packet is sent to the display control unit; After receiving the command packet, the display control unit will execute the cursor's upward movement on the screen according to the "move up" command; Assuming the current cursor coordinates are , after receiving the command packet, the cursor will be moved to , i.e., 10 pixel units upwards on the screen.

[0127] The display control unit executes the corresponding interaction action according to the command packet, such as: Cursor movement: According to the gesture category (e.g., move up, move down, move left, move right, etc.), control the cursor's movement on the screen to perform menu scrolling, page switching, etc.

[0128] Page scrolling: If a "swipe up" gesture is recognized, the system will perform the operation of scrolling the page upwards.

[0129] Click confirmation: If a "click" gesture is recognized, the system will perform the button click operation to confirm the selection or open the link.

[0130] Slide switching: if a "slide" gesture is recognized, the system will perform a slide operation on the screen, switching different pages or switching tabs.

[0131] For example: Suppose the user makes a "swipe up" gesture, which is determined to be a page scrolling gesture, and a command package is generated.

[0132] The command package is transmitted to the display control unit, which performs a page up scrolling operation according to the command, showing the user the previous page or more content.

[0133] The execution result of the interaction command is usually confirmed by visual or tactile feedback.

[0134] This feedback can inform the user that the operation has been successfully executed through screen display (such as cursor movement, button highlighting, etc.) or tactile vibration (such as a slight vibration).

[0135] For example: When the user performs a "click" gesture, the display control unit will not only perform the click operation, but also provide feedback to the user on the screen through the highlighting of the button, or through tactile feedback (such as a slight vibration) to indicate that the click has been completed.

[0136] Map the user's gesture operation to a specific interaction command and transmit the command to the display control unit through a bus or wireless protocol.

[0137] After receiving the command, the display control unit performs cursor movement, page scrolling, click confirmation, etc., and confirms the user's interaction operation through visual or tactile feedback. Embodiment two:

[0138] As shown in Figure 2 To solve the above technical problems, another technical solution adopted by the present application on the basis of embodiment one is: an eyeglass midair interaction system integrated with a gesture recognition module, comprising: A calibration mapping module is configured to obtain spatial calibration, integrate a TOF laser ranging module at the front end of the wearable eyeglass frame, the TOF laser ranging module is electrically connected with the master control unit of the eyeglass; the TOF laser ranging module scans the field of view in front of the eyeglass and collects reference depth data, at the same time, establishes a mapping relationship between the TOF measurement space and the target display screen through a calibration program, and stores the mapping relationship as a reference model for subsequent position projection; A frame difference extraction module is configured to continuously collect depth frame data based on the TOF laser ranging module at a predetermined sampling frequency, the master control unit compares the current depth frame with the reference depth data and extracts the foreground region according to the pre-set depth difference threshold, the foreground region is a candidate region where the hand or finger exists; The poly-tracing smoothing module is configured to perform spatial clustering and connected component analysis on the foreground region, calculate three-dimensional key points, and perform temporal tracking between continuous frames, while adopting filtering-based smoothing processing to suppress ranging noise and output stable three-dimensional position trajectories; The gesture recognition module is configured to extract trajectory features according to a predetermined time window and a spatial window based on the three-dimensional position trajectories, and map the trajectory to a predefined gesture category according to a preset rule or a trained classifier, so as to determine that the current gesture is up, down, left, right, click, or swipe. The command feedback module is configured to generate a command package corresponding to display screen interaction according to the gesture category and the position of the finger in the mapping relationship, and send the command package to a display control unit through a bus or wirelessly via a master control unit of the glasses, so that the display control unit performs corresponding cursor movement, page scrolling, click confirmation, and swipe switching interaction on the display device according to the received command, and simultaneously feeds back the execution result to the user in a visual or tactile manner.

[0139] For other details of the implementation of the technical solutions of the modules in the above embodiment system, refer to the description in the above embodiment of the method for air interaction of the glasses integrating the gesture recognition module.

[0140] It should be noted that each embodiment in the present specification adopts a progressive description manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between embodiments can be referred to each other. For system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment. Embodiment three:

[0141] As Figure 3 A structural schematic diagram of an electronic device is provided for the embodiments of the present disclosure. It shows a structural schematic diagram suitable for implementing the electronic device in the embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0142] As Figure 3 As shown, an electronic device includes a processor, a memory, and a communication interface. The memory stores a computer program. When the processor executes the computer program, the method for air interaction of the glasses integrating the gesture recognition module according to the embodiments of the present disclosure is implemented. The electronic device can exchange data with other devices or systems through the communication interface, realize real-time updating and sharing of drug information.

[0143] The processor in the electronic device described above is the core of the electronic device, and is responsible for executing computer programs stored in the memory to realize various functions of the paperless conference terminal intelligent interaction method. The processor can adopt a high-performance multi-core CPU or a dedicated chip to meet the needs of complex calculations and real-time processing. The memory is used to store the operating system, application programs, data and computer programs, etc. In this embodiment, the memory stores a computer program that implements the paperless conference terminal intelligent interaction method. The memory can be RAM, ROM, Flash memory or other types of non-volatile memory. The communication interface is used to connect the electronic device with other devices or networks to realize the transmission and exchange of data. In this embodiment, the communication interface supports multiple communication protocols and interface standards, such as Wi-Fi, Bluetooth, USB, Ethernet, etc., to meet the communication needs in different scenarios.

[0144] The detailed description of the present embodiment can refer to the corresponding description in the foregoing embodiments, which will not be repeated here. Embodiment Four

[0145] According to the computer readable storage medium of the present embodiment, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the functions of the above-mentioned eyeglasses hovering interaction method of the integrated gesture recognition module.

[0146] The above-mentioned computer readable storage medium includes but is not limited to: optical storage medium (for example: CD-ROM and DVD), magneto-optical storage medium (for example: MO), magnetic storage medium (for example: magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (for example: memory card) and media with built-in ROM (for example: ROM box).

[0147] The detailed description of the present embodiment can refer to the corresponding description in the foregoing embodiments, which will not be repeated here.

[0148] The above-mentioned only the preferred embodiments of the present application, and not to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application, should be included in the protection scope of the present application.

Claims

1. A method for air-based interaction with glasses integrating a gesture recognition module, characterized in that, Includes the following steps: Spatial calibration is obtained by integrating a TOF laser ranging module at the front end of the wearable glasses frame. The TOF laser ranging module is electrically connected to the main control unit of the glasses. The TOF laser ranging module scans the field of view in front of the glasses and collects reference depth data. At the same time, a mapping relationship between the TOF measurement space and the target display screen is established through a calibration program, and the mapping relationship is stored as a reference model for subsequent position projection. Based on the TOF laser ranging module, depth frame data is continuously collected at a predetermined sampling frequency. The main control unit compares the current depth frame with the reference depth data and extracts the foreground region according to the preset depth difference threshold. The foreground region is used as a candidate region for the existence of a hand or fingers. Spatial clustering and connected component analysis are performed on the foreground region to calculate three-dimensional key points and perform temporal tracking between consecutive frames. At the same time, filtering-based smoothing is used to suppress ranging noise and output a stable three-dimensional position trajectory. Based on the three-dimensional position trajectory, trajectory features are extracted according to a predetermined time window and spatial window, and the trajectory is mapped to a predefined gesture category according to a preset rule or a trained classifier to determine the current gesture as up, down, left, right, click, and swipe; Based on the gesture category and the finger position in the mapping relationship, a command package corresponding to the interaction with the display screen is generated and sent to the display control unit via bus or wirelessly through the main control unit of the glasses. The display control unit performs corresponding cursor movement, page scrolling, click confirmation and swipe switching interaction actions on the display device according to the received command, and at the same time feeds back the execution results to the user in a visual or tactile manner.

2. The interaction method according to claim 1, characterized in that, The mapping relationship is obtained by collecting a set of calibration samples at different viewing distances and angles during the initialization phase and fitting them using multinomial regression or bilinear interpolation methods.

3. The interaction method according to claim 1, characterized in that, The depth difference threshold is an adaptive threshold, which is adjusted in real time based on ambient light intensity, TOF reflectance readings, and historical baseline noise levels.

4. The interaction method according to claim 1, characterized in that, The filtering process employs a weighted Kalman filter or an exponential smoothing algorithm, and the process noise and measurement noise covariance parameters of the filter are dynamically updated through an adaptive strategy based on the current sampling stability. The three-dimensional key points include the hand centroid and fingertip key points.

5. The interaction method according to claim 1, characterized in that, The trajectory features include displacement, velocity, direction, and depth variation features, and the classifier has online incremental learning capabilities.

6. The interaction method according to claim 1, characterized in that, The command packet includes a unique instruction identifier, a gesture type identifier, the screen coordinates of the current finger mapping, and a timestamp. It is transmitted to the display control unit via a data frame format including a check bit on the communication link. After receiving the command, the display control unit performs the corresponding interactive action according to the current user interface context.

7. The interaction method according to claim 1, characterized in that, The method further includes: after the display control unit completes the interactive action, evaluating the correctness of the previous action recognition based on the user's subsequent actions or explicit feedback information, and using the evaluation results for local parameter adjustment or submitting them to the cloud for offline model optimization.

8. A glasses-based air interaction system integrating a gesture recognition module, applied to the glasses-based air interaction method integrating a gesture recognition module as described in any one of claims 1-7, characterized in that, include: The calibration mapping module is configured to acquire spatial calibration. A TOF laser ranging module is integrated at the front end of the wearable glasses frame. The TOF laser ranging module is electrically connected to the main control unit of the glasses. The TOF laser ranging module scans the field of view in front of the glasses and collects reference depth data. At the same time, it establishes a mapping relationship between the TOF measurement space and the target display screen through a calibration program, and stores the mapping relationship as a reference model for subsequent position projection. The frame difference enhancement module is configured to continuously acquire depth frame data at a predetermined sampling frequency based on the TOF laser ranging module. The main control unit compares the current depth frame with the reference depth data and extracts the foreground region according to a pre-set depth difference threshold. The foreground region serves as a candidate region for the presence of a hand or fingers. The trajectory smoothing module is configured to perform spatial clustering and connected component analysis on the foreground region, calculate three-dimensional key points and perform temporal tracking between consecutive frames, and at the same time use filter-based smoothing processing to suppress ranging noise and output a stable three-dimensional position trajectory. The gesture recognition module is configured to extract trajectory features based on the three-dimensional position trajectory according to a predetermined time window and spatial window, and map the trajectory to a predefined gesture category according to a preset rule or a trained classifier, and determine the current gesture as up, down, left, right, click, and swipe; The command feedback module is configured to generate a command package corresponding to the interaction with the display screen based on the gesture category and the position of the finger in the mapping relationship, and send it to the display control unit via bus or wirelessly through the main control unit of the glasses. The display control unit performs corresponding cursor movement, page scrolling, click confirmation and swipe switching interaction actions on the display device according to the received command, and at the same time feeds back the execution results to the user in a visual or tactile manner.

9. An electronic device, characterized in that, The device includes a processor, a memory, and a communication interface. The memory stores a computer program, and when the processor executes the computer program, it implements a glasses-based air interaction method with an integrated gesture recognition module as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a glasses-based air interaction method with an integrated gesture recognition module as described in any one of claims 1 to 7.

Images