Touch state detection method and system based on hand key points

Abstract

The invention relates to the field of gesture visual recognition, in particular to a touch state detection method and system based on hand key points. The method comprises the following steps: acquiring a hand image of a user, performing super-resolution reconstruction, and constructing a super-resolution image; performing hand feature point recognition and point cloud modeling on the super-resolution image, and constructing a real-time hand posture model; gesture action recognition is carried out on the real-time hand gesture model, and a plurality of complete gesture action results are recognized; performing gesture action virtual remodeling based on the plurality of complete gesture action results, and constructing gesture action virtual behavior data; and performing instant gesture state semantic analysis according to the gesture action virtual behavior data, and outputting an instant gesture semantic result. By efficiently and accurately recognizing the operation gesture of the user, the touch recognition response speed and the non-contact touch experience of the user are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of gesture visual recognition, and in particular to a touch state detection method and system based on hand key points. BACKGROUND

[0002] Traditional touch detection methods mainly rely on physical contact, such as capacitive screens, resistive screens, or pressure sensors, etc. These methods not only have high requirements for hardware conditions, but are difficult to adapt to non-contact or long-distance interaction scenarios, and also have obvious deficiencies in accuracy and stability in multi-user operation, complex environments (such as strong light, wet hands, etc.). In addition, traditional touch detection methods often cannot model and semantically understand the user's action process, limiting the implementation of higher-level intelligent interaction.

[0003] With the rapid development of computer vision, deep learning, and three-dimensional reconstruction technologies, non-contact gesture interaction based on image recognition has gradually become a research hotspot. In particular, the breakthroughs in hand key point recognition and depth vision fusion technology enable the system to accurately extract the spatial position information and motion trajectory of the user's hand, thereby enabling the perception and decoding of complex gesture actions. However, current such methods still face many challenges in touch action state detection and recognition. Gesture recognition models often only focus on the classification of static or standardized gestures, lack a deep understanding of continuous and goal-oriented actions, and are difficult to accurately recognize touch action states such as "click", "slide", "long press", etc. with specific interaction semantics. Secondly, limited by image resolution, lighting conditions, and pose occlusion, the recognition robustness and real-time performance of the system in complex environments are still not high. In addition, most existing solutions lack semantic analysis and state prediction capabilities for gesture behavior, and cannot achieve intelligent and dynamic response human-computer interaction mechanisms. SUMMARY

[0004] To solve the above technical problems, the present application provides a touch state detection method and system based on hand key points to solve at least one of the above technical problems.

[0005] To achieve the above purpose, the present application provides a touch state detection method based on hand key points, comprising the following steps: Step S1: Collecting a user's hand image, performing super-resolution reconstruction, and constructing a super-resolution image; Step S2: Performing hand feature point recognition and point cloud modeling on the super-resolution image to construct a real-time hand pose model; Step S3: Performing gesture action recognition on the real-time hand pose model to identify multiple complete gesture action results; Step S4: Performing gesture action virtual remodeling based on the multiple complete gesture action results to construct gesture action virtual behavior data; Step S5: Perform real-time gesture state semantic analysis based on the virtual behavior data of gesture actions, and output the real-time gesture semantic results.

[0006] This specification provides a touch state detection system based on hand key points, used to execute the touch state detection method based on hand key points as described above, including: The image optimization module is used to acquire images of the user's hand, perform super-resolution reconstruction, and construct a super-resolution image. The feature point recognition module is used to recognize hand feature points and model point clouds in super-resolution images to build a real-time hand pose model. The gesture recognition module is used to recognize gesture actions from a real-time hand posture model and identify multiple complete gesture action results. The virtual reshaping module is used to virtually reshape gestures based on multiple complete gesture results and construct virtual gesture behavior data. The gesture state output module is used to perform real-time gesture state semantic analysis based on virtual behavior data of gesture actions and output real-time gesture semantic results.

[0007] The beneficial effects of this invention are as follows: By using super-resolution technology to restore low-quality images, hand images achieve higher clarity and detail. Clearer images facilitate the extraction of more accurate key points, contours, and depth information in subsequent steps, especially for small parts such as fingertips and joints. High-quality visual information can still be obtained when using ordinary cameras or low-resolution sensors, improving the system's adaptability and versatility. Through point cloud data and feature point modeling, the user's true hand posture in three-dimensional space can be accurately reconstructed. It can capture every change in the user's hand movements in real time, providing dynamic and continuous input for gesture recognition. The posture model is more robust than a single-frame image, adapting to complex environments such as different angles, lighting changes, and partial occlusion. Constructing an accurate posture model significantly improves the accuracy and consistency of subsequent action recognition. It can recognize not only static gestures but also dynamic gesture sequences, such as swiping, clicking, and dragging. Based on dynamic posture sequences, it can effectively distinguish differences between similar gestures, reducing misidentification. The gesture recognition results are accurate and responsive, enhancing the user's smoothness and immersion when using the system. Multiple complete gesture results serve as "behavioral corpus," laying the foundation for subsequent semantic analysis. Unifying different users and actions into a unified virtual behavior data format facilitates system processing and analysis. It also facilitates visual verification, playback, and retraining of recognition results, improving system debuggability and interpretability. The action data, processed through virtual reshaping, is adaptable to different platforms and application scenarios, enhancing system versatility. Structured virtual behavior data is more suitable for inputting into neural network models for training and optimization. Low-level physical actions are mapped to high-level semantic commands such as "click," "zoom in," and "confirm," supporting natural language interfaces or task-driven operations. Semantic analysis results can be directly used to trigger system actions or control interaction processes, achieving true "what you see is what you get." The system not only identifies what the user is "doing" but also understands "why they are doing it," providing support for intelligent decision-making and prediction. Semantic-level feedback and understanding make human-computer interaction closer to human communication. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the steps of a touch state detection method based on key hand points according to the present invention. Figure 2 This is a detailed flowchart illustrating the implementation steps of step S1. Figure 3 This is a detailed flowchart illustrating the implementation steps of step S2; Figure 4 This is a flowchart illustrating the detailed implementation steps of step S3. Detailed Implementation

[0009] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0010] This application provides a touch state detection method and system based on key hand points. The execution entities of the method and system include, but are not limited to, mechanical equipment, data processing platforms, cloud server nodes, network upload devices, etc., which can be considered general computing nodes of this application. The data processing platform includes, but is not limited to, at least one of an audio-visual management system, an information management system, and a cloud data management system.

[0011] Please see Figures 1 to 4 This invention provides a touch state detection method based on key points of the hand, comprising the following steps: Step S1: Acquire images of the user's hand, perform super-resolution reconstruction, and construct a super-resolution image; Step S2: Perform hand feature point recognition and point cloud modeling on the super-resolution image to construct a real-time hand pose model; Step S3: Perform gesture recognition on the real-time hand pose model to identify multiple complete gesture results; Step S4: Based on multiple complete gesture action results, perform virtual reshaping of gesture actions to construct virtual gesture action behavior data; Step S5: Perform real-time gesture state semantic analysis based on the virtual behavior data of gesture actions, and output the real-time gesture semantic results.

[0012] In the embodiments of the present invention, see Figure 1 The diagram below illustrates the steps of a touch state detection method based on key hand points according to the present invention. In this example, the steps of the touch state detection method based on key hand points include: Step S1: Acquire images of the user's hand, perform super-resolution reconstruction, and construct a super-resolution image; In this embodiment, the original hand image sequence acquired by the camera device undergoes quality assessment preprocessing. Valid image frames with a resolution of at least 640×480 pixels are selected using an image sharpness detection algorithm. The valid image frames are then upscaled using the deep learning-based super-resolution reconstruction network ESRGAN, increasing the input image by a factor of 4 to 2560×1920 pixels while maintaining the integrity of image details. The super-resolution reconstructed images are then subjected to brightness equalization processing using the CLAHE (Contrast Limiting Adaptive Histogram Equalization) algorithm, with a cropping limit parameter of 2.0 and a grid size of 8×8, to eliminate uneven lighting on hand features. The impact on recognition is investigated. Bilateral filtering is applied to the brightness-optimized image for noise reduction, with spatial kernel standard deviation σd=5 and grayscale kernel standard deviation σr=50, suppressing image noise while maintaining edge clarity. An image sharpening enhancement algorithm is used to improve hand contour and texture details, employing the Laplacian operator for edge enhancement with an enhancement coefficient of 0.8. The output includes a 2560×1920 super-resolution image sequence I_super(t), an image quality assessment score Q_score, a brightness distribution histogram H_brightness, and an edge enhancement mask M_edge, providing high-quality image data for subsequent hand feature point recognition.

[0013] Step S2: Perform hand feature point recognition and point cloud modeling on the super-resolution image to construct a real-time hand pose model; In this embodiment, the output super-resolution image sequence I_super(t) is processed for hand detection using MediaPipe. A detection confidence threshold of 0.8 and a tracking confidence threshold of 0.7 are set to ensure the stability and accuracy of hand keypoint detection. 2D coordinate information of 21 key feature points is extracted from the hand detection results, including 1 point on the wrist, 5 points on the thumb, 4 points on the index finger, 4 points on the middle finger, 4 points on the ring finger, and 4 points on the little finger, forming a complete hand topology. The Perspective-n-Point (PnP) algorithm is applied to the 2D keypoint coordinates for depth estimation. Based on prior knowledge of hand anatomy, phalanx length constraints are set: thumb proximal phalanx length 32-38mm, index finger proximal phalanx length 39mm. -48mm, the proximal phalanx length of the middle finger is 43-54mm, and a 3D coordinate mapping relationship is constructed. Point cloud modeling is performed on the 3D coordinate data, and the Delaunay triangulation algorithm is used to construct the hand surface mesh. The mesh density is set to 100 vertices per square centimeter to ensure the geometric accuracy of the model. The key point coordinates are temporally smoothed by the Kalman filter algorithm, with process noise covariance Q=0.01 and observation noise covariance R=0.1 to eliminate detection jitter and outliers. The output includes 21 3D key point coordinate matrices P_3d(t), the hand point cloud model Cloud_hand, the key point confidence vector C_points, and the temporal smoothing parameters Smooth_params, to construct a stable and accurate real-time hand pose model.

[0014] Step S3: Perform gesture recognition on the real-time hand pose model to identify multiple complete gesture results; In this embodiment, a sliding window method is used to set the time window length to 30 frames (1 second) and the step size to 5 frames to extract the spatiotemporal feature vector of hand movements. An LSTM-Attention-based gesture recognition network is applied to these spatiotemporal features for action classification. The network structure contains 128 LSTM hidden units, 8 attention heads, and a learning rate of 0.001, capable of recognizing 15 basic gesture actions such as clicking, dragging, scaling, rotating, and stillness. The recognition results are evaluated for confidence, with an action recognition confidence threshold of 0.85. Actions below the threshold are marked as "unknown actions" to ensure the reliability of the recognition results. Temporal boundary detection is performed on single action segments, determining the complete action's temporal boundary through the speed change features at the start and end of the action. The action start threshold is set to a velocity change rate > 0.5 m / s², and the end threshold is set to < 0.1 m / s². Semantic annotation is performed on the detected complete actions to establish an action-semantic mapping dictionary, which includes multi-dimensional semantic attributes such as action type, duration, spatial range, and force intensity. The completeness of the identified actions is verified by an action quality assessment algorithm, and the evaluation indicators include action continuity (> 0.9), temporal consistency (> 0.85), and spatial rationality (> 0.8). The complete gesture action sequence A_complete(t), action semantic label L_semantic, action confidence distribution C_action, and action quality assessment score Q_action are output to provide an accurate action recognition basis for virtual reconstruction.

[0015] Step S4: Based on multiple complete gesture action results, perform virtual reshaping of gesture actions to construct virtual gesture action behavior data; In this embodiment, the output complete gesture sequence A_complete(t) is parametrically modeled, extracting key pose keyframes, motion trajectories, velocity curves, and acceleration distributions to establish a mathematical description model of the action. An inverse kinematics algorithm is applied to the parametric model to drive a virtual hand model, setting the virtual hand model to standard adult hand dimensions: hand length 180-190mm, hand width 75-85mm, and finger length ratio set according to ergonomic standards. The virtual hand model undergoes motion replay processing, generating smooth intermediate frames using an interpolation algorithm. Cubic spline interpolation is used to ensure the smoothness of the motion reconstruction, with a reconstruction frame rate set to 60fps. The reconstructed virtual action is then physically simulated and verified, using a rigid body dynamics model to check the physical rationality of the action. The gravitational acceleration g is set to... With a friction coefficient of 9.8 m / s² and μ=0.6, the virtual movements are ensured to conform to the laws of physics. The reconstructed movements are visualized using a virtual reality rendering engine to generate a high-fidelity 3D hand movement sequence with a rendering resolution of 1920×1080. The lighting model uses PBR (Physically Based Rendering), and the material parameters are set to skin roughness of 0.7 and subsurface scattering intensity of 0.3. The virtual reconstruction results are formatted to encode the movement data into standardized behavior descriptors, including multi-dimensional information such as timestamps, 3D coordinates, joint angles, and movement speed. The virtual gesture behavior data V_behavior, virtual movement sequence V_sequence, physical simulation parameters P_physics, and behavior descriptor D_behavior are output to construct a complete virtual gesture behavior dataset.

[0016] Step S5: Perform real-time gesture state semantic analysis based on the virtual behavior data of gesture actions, and output the real-time gesture semantic results.

[0017] In this embodiment, semantic features are extracted from the output virtual gesture behavior data V_behavior using a Transformer-based semantic encoder. The model parameters include 12 encoding layers, 768 hidden states, and 12 attention heads, enabling it to capture deep semantic features of gestures. A multimodal fusion network is applied to these semantic features for contextual understanding, combining spatial location information, temporal change patterns, and action intensity features. The fusion weights are set to 0.4, 0.35, and 0.25, respectively, to construct a comprehensive semantic representation vector. The semantic representation vector is then processed for intent classification. A softmax classifier is used to identify the user's interaction intent, which includes eight main categories such as navigation control, object manipulation, system interaction, and gesture expression. The classification accuracy requirement is >92%. The intent classification results are then evaluated for confidence and uncertainty is quantified using a Bayesian neural network to calculate predictions. To address uncertainty, an uncertainty threshold of 0.15 is set, with predictions exceeding this threshold marked as low confidence. A real-time semantic inference engine provides contextualized interpretation of gesture semantics, combining the current application scenario, user history, and environmental context to generate personalized semantic interpretation results. The semantic analysis results are formatted for output, generating structured semantic results containing information such as action type, semantic label, confidence score, timestamp, and spatial coordinates. A semantic feedback mechanism is established to evaluate the accuracy of semantic analysis through user interaction responses; if the feedback accuracy is >88%, the current parameters are maintained; otherwise, adaptive adjustment of model parameters is triggered. The system outputs instantaneous gesture semantic results S_semantic, semantic confidence distribution C_semantic, intent classification label L_intent, and semantic quality assessment report R_quality, achieving intelligent semantic understanding and accurate state recognition of hand touch actions.

[0018] In this embodiment, see Figure 2 The diagram below illustrates the detailed implementation steps of step S1. In this embodiment, the detailed implementation steps of step S1 include: The system continuously captures images of the user's hand using a touch-screen camera; global nonlinear gamma correction is then applied to the user's hand images to obtain a globally optimized brightness image. Image artifact noise is detected in the globally brightness optimized image, and artifact noise points are marked. Spatial distribution analysis of artifact noise points is performed to generate a noise characteristic distribution map; Calculate the frequency component of each noise point based on the noise characteristic distribution; Nonlocal noise filtering is performed based on the frequency components of the noise points to obtain a nonlocal filtered image. A super-resolution image is constructed by performing depthwise convolution super-resolution reconstruction on the non-locally filtered image.

[0019] In this embodiment, a high frame rate touch vision system continuously acquires images of the user's hand area. In actual system deployment, a high-sensitivity near-field infrared or visible light touch camera (such as the Intel RealSense D435, Orbbec Astra series, etc.) is used as the acquisition device. The sampling frequency is generally set to 30~60 fps, and the image resolution is set between 640×480 or 1280×720 to ensure a balance between temporal and spatial resolution. The camera is integrated into the touch plane or attached to the underside of a transparent glass panel to ensure that the image can simultaneously capture the minute contact dynamics between the finger and the touch surface. In addition, by adjusting the camera's mounting angle and focal length, it can be stably focused on the finger area, reducing the impact of external interference light sources on image quality. During data acquisition, a synchronous triggering mechanism is used to synchronize the recording of hand movements, coupled with background modeling and dynamic ROI (Region of Interest) selection mechanisms, to improve the clarity and stability of the hand area in the acquired images. All image data is cached in real time in a high-performance edge computing module for subsequent processing.

[0020] The original image, limited by lighting conditions, camera dynamic range, and skin tone differences, may suffer from overall low brightness or insufficient contrast. Global nonlinear gamma correction is employed to optimize image brightness, enhancing image details and improving overall visibility. Gamma correction is performed based on the nonlinear mapping function I_out = I_in^γ, where γ (the gamma coefficient) is typically set between 0.4 and 0.6. In experiments, multiple gamma values ​​(e.g., 0.4, 0.5, 0.6) were compared and evaluated. Ultimately, γ=0.5 yielded the best results, effectively enhancing texture details in bright areas without excessively increasing background noise. Gamma correction transforms the luminance channel (Y or V) in the YUV or HSV color space to preserve color distribution stability. The processed image exhibits a higher dynamic range, especially noticeable at the point of contact between the fingertips and the touch surface, providing clearer grayscale information for subsequent noise detection and keypoint localization. Global brightness optimization also effectively reduces the blurring caused by shadows in the original image. Artifact noise in images is typically caused by ambient light interference, glass reflection, lens smudges, or algorithmic gamma correction, often manifesting as regular stripes, blurred edges, or anomalous pixels. A hybrid artifact detection method based on gradient differences and local statistical features is used for processing. Specifically, the Sobel operator is used to extract the horizontal and vertical gradients of the image, combined with local contrast and entropy calculations to detect potential anomalous noise regions. For each pixel, a local window (e.g., 5×5) is defined, and its brightness variance and gradient abrupt change rate are statistically analyzed. If the values ​​exceed a set threshold (1.5 times the overall image mean in the experiment), it is marked as an artifact noise point. Furthermore, high-frequency filtering (e.g., the Laplacian operator) is used to further enhance details, thus more accurately distinguishing between real structural edges and artifact stripes. The detection process, combined with heatmap generation and threshold filtering algorithms, automatically marks artifact pixels and forms an artifact point layer. In the experiment, an average of approximately 300-500 artifact points were detected per frame, mainly concentrated near finger edges and highlights in the touch area, providing data support for subsequent noise space modeling.

[0021] After artifact point marking is completed, the system enters the spatial distribution modeling stage for noise. This process aims to extract the density, clustering pattern, and distribution trend of noise points in the image space, generating a noise distribution map with statistical characteristics. A method based on kernel density estimation (KDE) and two-dimensional Gaussian fitting is used to perform spatial clustering analysis on the marked points. First, the image is divided into fixed grid regions (e.g., 20×20 blocks), and the number and density of noise points in each region are counted and visualized. Next, mean-shift clustering or the DBSCAN algorithm is used to identify high-density noise regions in space and determine whether they exhibit regularity (e.g., linear stripes or concentrated blocks). The system further calculates the directional distribution of noise points (extracting the principal direction through principal component analysis (PCA)) to infer the directional characteristics of the noise source. Finally, the above information is combined to generate a two-dimensional heatmap or vector distribution map to express the overall spatial structure and clustering trend of noise in the image. This distribution map is not only used for the next step of frequency analysis but also provides a basis for locating noise sources (e.g., glass scratches, contact reflections).

[0022] Combining spatial distribution and grayscale variation characteristics, the frequency components of each noise point are extracted in the frequency domain. The specific processing flow is as follows: a local image patch is cropped from the neighborhood of each noise point (e.g., within an 11×11 or 15×15 pixel range), and transformed to the frequency domain using Discrete Fourier Transform (DFT) or Discrete Wavelet Transform (DWT). In the spectrogram, the main energy concentration region is extracted, and the dominant frequency of the noise point (the frequency corresponding to the maximum frequency amplitude) is calculated. Experiments analyzing multiple artifact samples revealed that most noise point frequencies are concentrated in the mid-to-high frequency region (20–80 Hz corresponding to the image pixel period), indicating strong local rapid change characteristics. To enhance the analysis effect, a multi-scale analysis strategy is adopted, extracting frequency features at multiple window sizes and performing weighted fusion. Furthermore, frequency direction information is also recorded to distinguish different types of noise (e.g., texture artifacts and spot noise). This frequency component data is finally organized into a vector field, combined with the spatial distribution map, providing key parameters for nonlocal filtering.

[0023] After obtaining the frequency features and spatial distribution, a non-local means (NLM) mechanism is used to denoise noisy regions in the image. Unlike traditional local filtering (such as Gaussian blur), NLM uses a weighted average of similar regions based on the repetitive structure within the image, effectively removing high-frequency noise while preserving edge details. In the processing flow, each noise point is treated as a center pixel, and multiple distant regions around it are extracted for similarity comparison, calculating the gray-level mean weighted value. The similarity measure considers the differences in frequency components and spatial orientation consistency (using cosine similarity and frequency difference weights), significantly improving filtering accuracy. In the experiment, a 3×3 search window and a 21×21 comparison window were used, and the filtering intensity parameter h=15 (for 8-bit images). This method is particularly suitable for hand texture regions with strong image structure repetition, preserving fingertip contours and contact boundary information while filtering out high-frequency artifacts. The output non-locally filtered image has a clearer texture background and fewer interference artifacts compared to the original image, providing high-quality input for subsequent keypoint recognition.

[0024] Image super-resolution reconstruction was performed using a deep learning-based convolutional neural network. The Enhanced Deep Super-Resolution Network (EDSR) was selected as the backbone network structure, composed of multiple stacked residual blocks, which effectively captures multi-scale feature information. The model accepts filtered low-resolution image input (e.g., 640×480) and outputs a high-resolution image (e.g., 1280×960), with a magnification factor of ×2. During training, supervised learning was performed using image pairs with real hand motion samples. The loss function was a combination of L1 loss and perceptual loss to optimize the image's performance in terms of pixel and perceptual quality. The super-resolution model ran on a GPU or edge TPU module, with an average inference time of less than 20ms per frame, meeting real-time processing requirements. In the reconstructed image, the fingertip contour is clearer, and the details of touch traces are richer, improving the accuracy of keypoint localization and the sensitivity of touch state change detection, laying a solid foundation for the final touch action recognition.

[0025] In this embodiment, see Figure 3 The diagram below illustrates the detailed implementation steps of step S2. In this embodiment, the detailed implementation steps of step S2 include: Perform hand edge detection on the super-resolution image and extract the hand edge contour line; Image thresholding is performed based on the hand edge contour to extract the hand boundary image; Visual recognition of deep feature points is performed on hand boundary images to mark multiple hand structural feature points; Calculate the coordinates of the plurality of hand structure feature points; perform image mapping and point cloud modeling based on the coordinates to construct a three-dimensional hand feature model; Hand boundary images are subjected to anatomical constraint analysis to extract hand constraint patterns; these patterns include the application of bone length ratio constraints, joint angle constraints, and finger bending patterns. Based on the constraints of the hand, the structural constraints of the 3D hand feature model are optimized to construct a real-time hand posture model.

[0026] In this embodiment, the multi-scale Canny edge detection algorithm is used for edge detection. This algorithm is robust to noise and can effectively separate hand boundaries in complex backgrounds. Specifically, the image is first Gaussian filtered to smooth noise components (Gaussian kernel size set to 5×5, standard deviation σ=1.4). Then, the Canny algorithm is used with dual thresholds (low threshold=50, high threshold=150) to process image gradient information and extract high-response edge segments. During the experiment, to improve the closure of hand edges, morphological operations (such as dilation and closing operations) are introduced to post-process the edge map, eliminating broken edges and boundary noise. The final output hand edge contour line shows complete palm and finger boundaries, providing spatial discrimination criteria for contact points and non-contact areas, especially performing well for dynamic boundary capture in multi-finger operations and swipe touch. A dynamic Region of Interest (ROI) is constructed within the edge region, and the Otsu method is applied based on the brightness histogram distribution within the region to determine the local optimal segmentation threshold. In the experimental setup, the threshold is generally between 90 and 130 for image brightness, automatically adjusting according to ambient light. Simultaneously, to improve the segmentation integrity of the finger region, a conditional region growing algorithm is employed, incorporating similar grayscale regions adjacent to the hand contour into the hand mask. The segmentation results are then processed using connected component analysis to remove small pseudo-regions, retaining only connected regions larger than 1000 pixels as the main hand region. The output "hand boundary image" is a binary image, with the hand region displayed as white and the rest as black, providing clear structural input for subsequent feature point recognition and pose modeling.

[0027] Keypoint recognition is achieved using deep learning visual models, with common models being convolutional neural network systems based on HRNet (High-Resolution Network) or MediaPipe Hand modules. These models perform end-to-end prediction of images through fully convolutional networks, outputting a heatmap of coordinates for 21 standard hand keypoints. The input is a combination of the hand boundary image and the original grayscale image, and the output is a heatmap of the response map for each keypoint. After Gaussian kernel processing, the maximum value is extracted from the response map to obtain the keypoint coordinates. In the experiments, the prediction error for each keypoint was controlled within ±3 pixels, and the model showed high adaptability to changes in finger bending and palm posture. Especially under touch conditions, the keypoint responses in areas such as the fingertips and fingertips were stable, providing crucial support for the detection and tracking of dynamic touch behaviors (clicking, swiping, pressing). Combining binocular camera or TOF (Time-of-Flight) depth maps, the depth information of each keypoint was sampled to form (x, y, z) 3D coordinates. Subsequently, the complete hand skeleton structure was reconstructed through 3D interpolation and spline fitting, generating a preliminary point cloud model. In the experimental setup, the point cloud consisted of 21 key points connected to form a hand topology, with the connections between points defined according to anatomical skeletal connection rules. Using an Intel RealSense camera, depth accuracy reached ±1mm. The point cloud coordinates were mapped to the screen touch coordinate system through spatial transformation, achieving joint spatial-image modeling. Furthermore, to improve the stability of the point cloud model, the system performed temporal smoothing processing (such as Kalman filtering) on ​​the distance changes between points to avoid model jitter caused by detection errors. The final 3D hand model served as the foundational data structure for dynamic pose analysis, laying the framework for subsequent gesture recognition and motion decoding.

[0028] The system analyzes the bone length ratios between key points, setting standard length ratios based on the average adult skeletal model. For example, the finger joint ratio is set to proximal:middle:distal = 1:0.75:0.6, with an allowable error of ±15%. Secondly, joint rotation angles are constrained, especially for joints like the DIP, PIP, and MCP, limiting their natural range of motion to 0°~90° to avoid anatomically impossible reverse bending. Furthermore, the system incorporates a finger bending pattern model, fitting the bending trajectory based on the fingertip path changes between consecutive frames in a time-series image, and physically constraining its velocity and acceleration. This constraint mechanism not only improves the physical plausibility of the model's movements but also effectively suppresses pose drift caused by anomaly identification. Prior hand anatomy datasets (such as the MANO model) are used for parameter fitting in experiments, significantly enhancing the system's adaptability to individual differences. A nonlinear optimization method based on energy minimization is used, with the objective function consisting of: bone length fidelity loss, angle constraint loss, point cloud-image error loss, and temporal continuity loss. The optimization algorithm employs L-BFGS or Gauss-Newton iterations, with the spatial coordinates of all 21 key points as the control variables. In actual deployment, to meet real-time requirements, the optimization steps are accelerated on GPUs or edge AI chips, with processing time per frame controlled within 15ms. The optimization output is a consistent, coherent 3D hand skeleton pose map, with precise synchronization of the spatial position and angle information of each node. This real-time pose model supports gesture recognition, action segmentation, and behavioral intent analysis, dynamically reflecting the user's fingertip trajectory, contact position, and multi-finger coordination state during touch operations. It is a key supporting structure for achieving high-precision touch action state recognition.

[0029] In this embodiment, reference Figure 4 The diagram below illustrates the detailed implementation steps of step S3. In this embodiment, the detailed implementation steps of step S3 include: Dynamic optical flow tracing is performed on each feature point of the real-time hand posture model to extract the optical flow trajectory of all feature points; Calculate the dwell time of feature points based on the optical flow trajectory; Based on the dwell time of feature points, the rhythm of the action is analyzed to extract the pause points and completion points of the action. Calculate the moving speed and moving amplitude of the optical flow trajectory to generate trajectory features; The system performs complete gesture recognition based on trajectory features, pause points, and completion points, identifying multiple complete gesture results.

[0030] In this embodiment, based on the constructed 3D hand pose model, precise optical flow tracking is performed on each key point using temporal continuity to obtain the complete temporal trajectory of the key points. The Lucas-Kanade (Pyramidal LK) method in dense optical flow is employed. This algorithm is suitable for small-range continuous motion and has good temporal stability. Position matching is performed between frames for each feature point (such as fingertip, knuckle, and base of the finger). The algorithm searches for the optimal matching position between the previous and current frames based on the minimum error estimation of the grayscale window. To improve 3D tracking accuracy, after initial tracking at the 2D image level, the algorithm combines depth data (such as Z-axis information from TOF or a stereo camera) for Z-axis position alignment to achieve 3D dynamic tracking. In the experiment, the window size was set to 15×15 pixels, the number of pyramid layers was set to 3, the maximum number of iterations was 30, and the displacement tolerance threshold was set to 0.01. All feature points are tracked in real time at 30 frames per second, and their respective movement trajectory sequences are constructed on the time axis. This trajectory data provides the core foundation for dynamic changes in subsequent gesture recognition.

[0031] For any feature point, if the 3D spatial displacement within several consecutive frames is less than a set threshold (e.g., less than 0.5 mm per frame), the point is determined to be stationary. The number of stationary frames is then counted to calculate the corresponding dwell time (e.g., at 30 fps, 10 frames of stillness equals 333 ms). The system sets a minimum threshold of 5 frames of stillness to prevent brief jitters from being mistaken for pauses. For feature points with high-frequency dynamics (e.g., fingertips), the system also incorporates a speed threshold (below 0.01 m / s) as an auxiliary criterion to enhance the robustness of stillness detection. Experimental data shows that each complete touch action on average includes at least one pause of 0.3 to 1 second, especially in clicks and long presses, where the dwell time is highly correlated with the intended action. Therefore, this dwell time is not only an important indicator for judging the completion status of touch actions but also provides necessary time parameters for action rhythm analysis. By statistically analyzing the dwell time sequence in the feature point trajectory, the system can further deduce the rhythmic characteristics of the action, thereby identifying "pause points" (intention transitions or waiting) and "completion points" (action termination or confirmation). Action rhythm analysis is performed using a joint temporal-spatial model. First, the salience of several static states is determined by jointly considering dwell time and speed changes. For example, a pause time greater than 0.4 seconds with a significant decrease in displacement between frames is identified as an "action pause point." If a continuous static state occurs after an action and remains stationary on the touch surface for a period of time (e.g., more than 0.8 seconds), it is identified as a "completion point." Furthermore, the system supplements the detection results by identifying inflection points and extreme points of speed changes in the trajectory, ensuring the capture of key nodes such as swipe-to-stop and tap-to-confirm. The rhythm analysis accuracy reaches over 92%, effectively separating sub-stages in gestures (e.g., the "move-pause-confirm" three-stage structure). This rhythm information is an indispensable temporal feature input for subsequent action intent determination and multi-stage gesture classification (e.g., double-tap, long-press).

[0032] The movement speed (Speed) is calculated as the change in three-dimensional coordinates per unit time, using the formula: v = Δd / Δt, where Δd is the Euclidean distance between consecutive frames, and Δt is the time interval (usually 1 / 30 of a second). Next, the movement amplitude (Amplitude) is calculated, which is the spatial distance between the maximum and minimum points in the entire trajectory. To extract features with strong stability, the system segments the trajectory and calculates the average speed, maximum speed, speed change slope, and acceleration trend for each segment. Simultaneously, to reflect the overall structure of the trajectory, the system also calculates geometric features such as trajectory curvature, total path length, and frequency of direction changes. Experiments revealed significant differences in trajectory features among different gestures: for example, swiping actions have higher path lengths and average speeds, while tapping and pressing have shorter trajectories, lower speeds, and smoother changes. All these trajectory features are uniformly encoded into vector form, serving as high-dimensional input vectors for the gesture classification model, providing numerical support for action recognition. All constructed spatiotemporal trajectory features, rhythmic information (pause points and completion points), trajectory amplitude, and speed features are input into the gesture recognition model for complete action recognition and classification. For model selection, a hybrid deep network combining time-series and graph neural networks is employed, with a typical structure being a fusion architecture based on temporal convolutional networks (TCN) and graph convolutional networks (GCN). TCN is responsible for extracting the temporal evolution features of the trajectory, while GCN is responsible for modeling the spatial topological relationships between key hand points. In the experimental setup, the model recognized 10 typical touch gestures, including: click, long press, swipe, two-finger zoom, double-tap, finger opening / closing, fingertip rotation, lift, middle finger swipe, and five-finger spread. The training data comes from a labeled high-frame-rate touch interaction dataset, with an average of over 500 samples per class. The model ultimately achieved a recognition accuracy of 94.7% on the test set. Each gesture recognition process takes no more than 20ms, making it suitable for real-time recognition and feedback scenarios. The final system output consists of a complete action type label and its timestamp, forming the core decision-making unit for touch state recognition, providing accurate input for UI responses, virtual control, and other applications.

[0033] In this embodiment, step S4 includes the following steps: Deep analysis of finger function is performed based on multiple complete hand gestures to obtain finger function classification; the finger function classification includes dominant fingers, auxiliary fingers, and stabilizing fingers. Multi-finger gesture collaboration is analyzed based on finger function classification to obtain multi-finger gesture interaction modes; Action state recognition is performed based on multi-finger gesture interaction mode, and gesture action virtual reshaping is carried out to construct virtual gesture action behavior data.

[0034] In this embodiment, the five fingers are functionally classified into dominant fingers, auxiliary fingers, and stabilizing fingers by statistically analyzing the frequency of functional participation, motion characteristics, and spatial dominance of each finger in different gestures. Specifically, the method involves extracting indicators such as trajectory length, peak speed, maximum range of motion, and touch participation ratio from key points of each finger (the bone nodes corresponding to the thumb, index finger, middle finger, ring finger, and little finger out of 21 structural points) in multiple gestures. Dominant fingers are defined as those that play a leading role in active movement and contact with the touch surface in multiple actions, typically exhibiting the longest trajectory length, fastest speed, and fewest pauses (e.g., the thumb and index finger). Auxiliary fingers have some cooperative movement or slight contact functions; for example, in two-finger zooming, the middle or ring finger may participate in the action but does not play a primary operational role. Stabilizing fingers typically have very small range of motion, serving to stabilize the palm or maintain posture balance; for example, the little or ring finger remains stationary in most operations. In the experiment, analysis of over 5000 gesture samples revealed that, on average, only 2.1 fingers were dominant or auxiliary fingers in each type of action, with the rest being stable fingers, forming a stable functional classification distribution map. This classification provides high-level semantic support for subsequent multi-finger collaborative mode modeling. The dynamic relationships exhibited by multiple fingers in specific gestures were analyzed to construct a "multi-finger gesture collaborative interaction mode." This process is based on time-aligned finger trajectories, integrating their functional attribute labels (dominant / auxiliary / stable), and employing a graph structure modeling method to represent each action instance as a "Finger Interaction Graph." In this graph, each node represents a finger, with attributes including movement amplitude, trajectory direction, and participation status; edges represent the collaborative relationship between two fingers (such as contact time synchronization, relative movement direction angle, distance change trend, etc.). Through analysis of the finger edges, the system can identify collaborative operation types, such as synchronous collaboration (e.g., two-finger zoom), asynchronous compensation (e.g., thumb guidance, middle finger fine-tuning), or non-collaboration (e.g., click + fixation). K-means and graph clustering algorithms were used to cluster the finger-to-finger relationship graph, yielding a series of typical multi-finger interaction patterns, such as "two-finger symmetrical swipe," "primary / secondary finger switching click," and "three-finger simultaneous zoom." In the experiment, 12 stable collaborative patterns were constructed, covering more than 90% of common touch behavior multi-finger interaction scenarios. This interaction pattern analysis provides a structured intermediate representation for further decoding of actions into intentions in the system.

[0035] By matching the identified interaction patterns with an established action semantic database, the system identifies the current state stage of the gesture (such as "starting interaction," "in execution," "completed," or "interrupted"), thus forming a semantic annotation for the entire lifecycle of the action. The system models each gesture process using a State Transition Graph, determining the state transition based on the temporal sequence of the pattern, key pauses, and change trends. For example, in the "long press + swipe" gesture, the system first identifies the stable pressing state, then the slow swiping stage, and finally determines the "completed state" when the swiping ends. After identifying the complete action, the system further reconstructs the process using a standard virtual behavior format, generating "virtual gesture action behavior data." This involves reconstructing each gesture action in a parameterized form as a six-dimensional behavior data structure, including action category, start time, duration, participation index combination, path vector, and force estimation. This virtual reconstruction can serve various purposes, such as virtual gesture restoration in VR / AR environments, behavior-driven interfaces for human-computer collaboration models, or for building a training gesture behavior database.

[0036] In this embodiment, the specific steps of step S5 are as follows: Scene depth segmentation and contextual semantic awareness are performed based on user hand images to generate a contextual semantic awareness map; The virtual behavior data of gesture actions is input into the context semantic perception graph, and real-time gesture state semantic analysis is performed to output real-time gesture semantic results. A post-accuracy evaluation is performed on the instantaneous gesture semantic results to obtain a semantic recognition accuracy evaluation report; Multi-objective self-optimization learning is performed based on the accurate evaluation report of semantic recognition to generate self-optimization learning parameters; The context semantic perception graph is optimized by knowledge distillation based on self-optimizing learning parameters, and an online touch gesture state detection optimization task is performed.

[0037] In this embodiment, a multi-scale semantic segmentation model based on deep convolutional neural networks, such as DeeplabV3+ or HRNet-Semantic, is employed. Combining image and depth data inputs, refined scene semantic segmentation is achieved. The model input is an RGB-D image, where RGB represents a high-resolution image of the hand and D is a simultaneously captured depth map. Through pixel-level classification output, the system can distinguish multiple semantic categories such as "screen area," "button area," "background desktop," "interactive controls," and "icon targets." To ensure boundary clarity, the model uses dilated convolution to enhance multi-scale context awareness and introduces CRF (Conditional Random Field) post-processing for edge refinement. The segmentation results are organized into a "contextual semantic awareness graph," where each graph node represents a semantic object region (such as a control, window, or menu), and edges represent spatial relationships and reachable paths. The system projects finger touch points, movement trajectories, action times, and path directions from virtual behavior data into a semantic graph space and matches them with graph nodes. By calculating spatial overlap, temporal fit, and action path similarity (e.g., IoU overlap rate > 0.5, temporal window alignment difference < 300ms), the semantic object (e.g., a button or control) to which the action points is determined. Subsequently, based on the functional attributes of nodes in the graph (e.g., "submit button," "scroll bar," "folder icon") and the gesture type (e.g., click, swipe, long press), the system calls a semantic reasoning rule base for state recognition. For example, the combination of "click + submit button" is recognized as "confirmation intent," and "two-finger expansion + image area" is recognized as "zoom in intent." This processing flow is based on rule-based semantic mapping and a lightweight knowledge graph reasoning module (e.g., RDF + SPARQL query), with an average response time controlled within 15ms. The output is a structured "real-time gesture semantic result," in the format: [target object, operation type, interaction state], used to drive UI responses or behavior recording.

[0038] After each interaction cycle, the system compares the recognition results with the user's actual feedback or preset interaction logic. For example, in a UI system, if the user performs the "click submit" action but the interface does not change its state, it is considered a semantic reasoning bias. The system uses multiple metrics such as accuracy, precision, recall, and F1 score for statistical analysis. Using actual user operation logs as reference labels, a window overlap evaluation mechanism is adopted, with an error tolerance of ±3 frames and a spatial offset tolerance of ±5 pixels. In addition, the system records the types of reasoning errors (such as "target mismatch," "semantic offset," and "recognition delay") and analyzes the causes, generating a complete "Semantic Recognition Accuracy Evaluation Report." The report includes: a trend chart of recognition accuracy, the distribution of recognition difficulty for various gestures, and the current semantic graph response accuracy distribution, generated on average once per minute to adapt to real-time feedback scenarios. This report provides quantitative evaluation basis for the system's subsequent self-optimization learning module and supports dynamic adjustment of inference rules or model parameters. The system employs reinforcement learning and a multi-objective optimization framework to improve recognition accuracy, response speed, and model stability. First, it establishes a gesture semantic recognition state-action space. The state consists of the current semantic graph state, input features, and prediction results, while the actions are parameter adjustment strategies (such as graph matching threshold, sliding window length, and trajectory prediction strategy). The reward function is a combination of the improvement in recognition accuracy, the decrease in prediction latency, and resource consumption weights. Algorithms such as Deep Q-Network (DQN) or Proximal Policy Optimization (PPO) are used for policy learning. Each iteration includes the semantic recognition execution and feedback records of hundreds of gesture instances. The learning result is a set of optimal parameter configurations, i.e., a "self-optimizing learning parameter set," which includes a semantic matching weight matrix, rule triggering threshold, and trajectory smoothing adjustment factor. This parameter set can be updated every 10 minutes to ensure the system adapts to changes in user operation styles and scene complexity, supporting the construction of a continuously evolving gesture semantic parsing system.

[0039] Knowledge distillation is used to optimize the contextual semantic graph to enhance the model's generalization ability and operational efficiency. Knowledge distillation technology achieves a balance between inference accuracy and speed by transferring knowledge from high-performance models (such as complex inference models containing all semantic and behavioral features) to lightweight models (such as graph inference modules suitable for real-time edge detection). In implementation, a teacher model is first constructed, containing the complete semantic graph structure and all historical training data inference paths; the student model is a simplified version, retaining only key semantic nodes and high-frequency gesture trajectory patterns. The distributions of the outputs of both models are aligned and trained using KL divergence or cross-entropy loss functions, adjusting the student model's structure and parameters to closely approximate the teacher model's inference results. The experiment uses a temperature coefficient T=3 for distillation smoothing, with approximately 2000 training cycles. The final student model's recognition speed improves by about 40%, while the inference accuracy decreases by no more than 1.5%. The distilled and optimized semantic graph is deployed to touch-interactive terminals or edge servers to support high-speed detection and response to online touch gesture states. Once this stage is completed, the system can realize an adaptive touch interaction mechanism with semantic understanding as its core, achieving a closed-loop intelligent recognition system that goes from input recognition to semantic feedback to model optimization.

[0040] In this embodiment, the specific steps for performing scene depth segmentation and contextual semantic awareness based on the user's hand image to generate a contextual semantic awareness map are as follows: Scene depth segmentation is performed based on the user's hand image to obtain scene depth segmentation results; the scene depth segmentation results include the foreground hand region, mid-ground interactive objects, and background environment layers; Real-time scene interaction analysis is performed based on the scene depth segmentation results to obtain the current scene interaction features; Contextual semantic awareness is performed on the interaction features of the current scene to generate a contextual semantic awareness map.

[0041] In this embodiment, a deep learning segmentation method integrating multimodal inputs is employed: the input data is a four-channel image (RGB-D) composed of an RGB image and a depth map (D map). Lightweight semantic segmentation networks such as BiSeNet V2 or EfficientPS are used to improve segmentation speed while ensuring edge detection accuracy. During the training phase, publicly available hand interaction datasets (such as EgoHands and HOF) and a self-built multi-scene hand interaction image set are used for joint training. The category labels are divided into "hand region", "interaction object region", and "background region". Cross-entropy and Dice loss function are used for joint optimization during training. The segmentation network output is a pixel-level classification map, which is further adjusted through depth thresholding segmentation technology: the hand region generally has the smallest depth value and is closest to the camera; the interaction object is usually in the mid-range depth range (20~50cm); and the background region corresponds to the maximum depth layer (>80cm). The final output "scene depth segmentation result" is a spatial three-dimensional distribution map that supports simultaneous recognition of the three regions and is expressed with different channel encoding structures to support the spatial determination basis for subsequent interaction analysis tasks. The system extracts regional attributes from mid-range interactive objects in the segmentation results, including the target region's bounding box, center point coordinates, area, and spatial relative position with the hand (relative depth difference, relative pixel distance, etc.). A shortest path algorithm based on the geometric center point and fingertips (such as the Fast Marching Method) is used to estimate the proximity between the fingertips and the object region. When the proximity is less than a set threshold (e.g., 8 pixels) and the duration exceeds a set value (e.g., 10 frames), the system identifies the current target as an object undergoing "intent contact." Furthermore, the system performs optical flow analysis (using the Dense Optical Flow or RAFT module) on the dynamic information of the foreground hand region, extracting dynamic features such as finger movement direction, speed, and dwell time to further determine whether the interaction action is a click, swipe, press, or hold. The system comprehensively encodes this information to form a "current scene interaction feature" vector, including key dimensions such as [distance between hand and object], [frequency of object interaction], [direction of fingertip trajectory], [finger pressure estimation (based on displacement velocity)], and [finger aggregation degree], laying the feature foundation for subsequent semantic mapping. The average processing latency is no more than 40ms, which meets the timeliness requirements of real-time interactive analysis.

[0042] The system semantically aligns the interactive features extracted in the previous stage with object categories. Using a pre-defined object semantic database (such as "buttons," "sliders," "image areas," and "text boxes"), combined with object labels identified by the image recognition module (YOLOv7 or DETRTransformer), semantic values ​​are assigned to the current mid-scene interactive objects. Next, based on the current interaction relationship between the hand and the object, the system maps the interaction scene into a "semantic node graph" structure. Each identified object is treated as a node, recording attributes such as object ID, spatial location, current interaction state, interaction frequency, and relationship with the finger. The system further constructs edge relationships between the semantic nodes according to spatial adjacency and interaction sequence logic, forming a graph structure containing hierarchical relationships and interaction paths. The final output is a "contextual semantic awareness graph," which can be implemented based on a graph database (such as Neo4j) or a lightweight knowledge graph representation framework (such as RDF-Turtle), possessing query, update, and reasoning capabilities. This graph supports dynamic expansion. When a user enters a new scene or creates a new interactive object, new nodes can be automatically generated and associated with semantic attributes, thereby realizing real-time dynamic semantic modeling and behavior understanding, providing core semantic support for gesture intent recognition and virtual interaction reconstruction.

[0043] In this embodiment, a touch state detection system based on hand key points is provided, used to execute the touch state detection method based on hand key points as described above, including: The image optimization module is used to acquire images of the user's hand, perform super-resolution reconstruction, and construct a super-resolution image. The feature point recognition module is used to recognize hand feature points and model point clouds in super-resolution images to build a real-time hand pose model. The gesture recognition module is used to recognize gesture actions from a real-time hand posture model and identify multiple complete gesture action results. The virtual reshaping module is used to virtually reshape gestures based on multiple complete gesture results and construct virtual gesture behavior data. The gesture state output module is used to perform real-time gesture state semantic analysis based on virtual behavior data of gesture actions and output real-time gesture semantic results.

[0044] Therefore, the embodiments should be considered as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalents of the application are intended to be included within the invention.

[0045] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein are implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A touch state detection method based on key hand points, characterized in that, Includes the following steps: Step S1: Acquire images of the user's hand, perform super-resolution reconstruction, and construct a super-resolution image; Step S2: Perform hand feature point recognition and point cloud modeling on the super-resolution image to construct a real-time hand pose model; Step S3: Perform gesture recognition on the real-time hand pose model to identify multiple complete gesture results; Step S4: Based on multiple complete gesture action results, perform virtual reshaping of gesture actions to construct virtual gesture action behavior data; Step S5: Perform real-time gesture state semantic analysis based on the virtual behavior data of gesture actions, and output the real-time gesture semantic results.

2. The touch state detection method based on key hand points according to claim 1, characterized in that, The specific steps of step S1 are as follows: The system continuously captures images of the user's hand using a touch-screen camera; global nonlinear gamma correction is then applied to the user's hand images to obtain a globally optimized brightness image. Image artifact noise is detected in the globally brightness optimized image, and artifact noise points are marked. Spatial distribution analysis of artifact noise points is performed to generate a noise characteristic distribution map; Calculate the frequency component of each noise point based on the noise characteristic distribution; Nonlocal noise filtering is performed based on the frequency components of the noise points to obtain a nonlocal filtered image. A super-resolution image is constructed by performing depthwise convolution super-resolution reconstruction on the non-locally filtered image.

3. The touch state detection method based on key hand points according to claim 1, characterized in that, The specific steps of step S2 are as follows: Perform hand edge detection on the super-resolution image and extract the hand edge contour line; Image thresholding is performed based on the hand edge contour to extract the hand boundary image; Visual recognition of deep feature points is performed on hand boundary images to mark multiple hand structural feature points; Calculate the coordinates of the plurality of hand structure feature points; perform image mapping and point cloud modeling based on the coordinates to construct a three-dimensional hand feature model; Hand boundary images are subjected to anatomical constraint analysis to extract hand constraint patterns; these patterns include the application of bone length ratio constraints, joint angle constraints, and finger bending patterns. Based on the constraints of the hand, the structural constraints of the 3D hand feature model are optimized to construct a real-time hand posture model.

4. The touch state detection method based on key hand points according to claim 1, characterized in that, Step S3 is as follows: Dynamic optical flow tracing is performed on each feature point of the real-time hand posture model to extract the optical flow trajectory of all feature points; Calculate the dwell time of feature points based on the optical flow trajectory; Based on the dwell time of feature points, the rhythm of the action is analyzed to extract the pause points and completion points of the action. Calculate the moving speed and moving amplitude of the optical flow trajectory to generate trajectory features; The system performs complete gesture recognition based on trajectory features, pause points, and completion points, identifying multiple complete gesture results.

5. The touch state detection method based on key hand points according to claim 1, characterized in that, The specific steps of step S4 are as follows: Deep analysis of finger function is performed based on multiple complete hand gestures to obtain finger function classification; the finger function classification includes dominant fingers, auxiliary fingers, and stabilizing fingers. Multi-finger gesture collaboration is analyzed based on finger function classification to obtain multi-finger gesture interaction modes; Action state recognition is performed based on multi-finger gesture interaction mode, and gesture action virtual reshaping is carried out to construct virtual gesture action behavior data.

6. The touch state detection method based on key hand points according to claim 1, characterized in that, The specific steps of step S5 are as follows: Scene depth segmentation and contextual semantic awareness are performed based on user hand images to generate a contextual semantic awareness map; The virtual behavior data of gesture actions is input into the context semantic perception graph, and real-time gesture state semantic analysis is performed to output real-time gesture semantic results. A post-accuracy evaluation is performed on the instantaneous gesture semantic results to obtain a semantic recognition accuracy evaluation report; Multi-objective self-optimization learning is performed based on the accurate evaluation report of semantic recognition to generate self-optimization learning parameters; The context semantic perception graph is optimized by knowledge distillation based on self-optimizing learning parameters, and an online touch gesture state detection optimization task is performed.

7. The touch state detection method based on key hand points according to claim 6, characterized in that, The specific steps for performing scene depth segmentation and contextual semantic awareness based on the user's hand image to generate a contextual semantic awareness map are as follows: Scene depth segmentation is performed based on the user's hand image to obtain scene depth segmentation results; the scene depth segmentation results include the foreground hand region, mid-ground interactive objects, and background environment layers; Real-time scene interaction analysis is performed based on the scene depth segmentation results to obtain the current scene interaction features; Contextual semantic awareness is performed on the interaction features of the current scene to generate a contextual semantic awareness map.

8. A touch state detection system based on key hand points, characterized in that, The method for performing touch state detection based on hand key points as described in claim 1 includes: The image optimization module is used to acquire images of the user's hand, perform super-resolution reconstruction, and construct a super-resolution image. The feature point recognition module is used to recognize hand feature points and model point clouds in super-resolution images to build a real-time hand pose model. The gesture recognition module is used to recognize gesture actions from a real-time hand posture model and identify multiple complete gesture action results. The virtual reshaping module is used to virtually reshape gestures based on multiple complete gesture results and construct virtual gesture behavior data. The gesture state output module is used to perform real-time gesture state semantic analysis based on virtual behavior data of gesture actions and output real-time gesture semantic results.

Images