Gesture interaction system for multi-dimensional gesture instruction recognition

Abstract

This invention relates to the field of hand skeleton topological constraints and dynamic gesture recognition, and discloses a gesture interaction system for multi-dimensional gesture command recognition, including: coordinate acquisition, topology modeling, variation coefficient extraction, jump arbitration, coordinate prediction, and command conversion modules. The coordinate acquisition module acquires the coordinates of hand joints; the topology modeling module establishes a skeletal topology map based on the skeletal connection relationships; the variation coefficient extraction module calculates the rate of change of topological side lengths based on the coordinate sequence to generate dynamic variation coefficients; the jump arbitration module compares the variation coefficients with a consistency threshold to determine non-physical jump points; the coordinate prediction module performs logical prediction for jump points; and the command conversion module converts the sequence into control commands. This invention converts coordinates into dynamic strain coefficients through topology modeling, eliminating the interference of user size differences on feature extraction; the system uses skeletal topological constraints to perform logical compensation for jump points, maintaining the physical self-consistency of the feature flow.

Description

Technical Field

[0001] This invention relates to the field of hand skeleton topological constraints and dynamic gesture recognition technology, and in particular to a gesture interaction system for multidimensional gesture command recognition. Background Technology

[0002] Currently, gesture interaction, as an important component of human-computer interaction, collects hand movement data through visual sensors to achieve non-contact control logic. It uses sensors to extract the three-dimensional spatial coordinates of hand joints to construct feature vectors representing gesture movements and matches them with preset command templates. Such solutions can complete basic command output in a controlled laboratory environment. However, in real dynamic interaction scenarios, existing technologies face physical constraints such as scale sensitivity and sampling noise interference. Since gesture feature extraction is highly dependent on the absolute spatial coordinates of joints, when there are differences in palm size between different users or changes in the relative displacement between the user and the sensor, the extracted feature vector values ​​will produce non-linear shifts. In addition, drastic changes in light and shadow or hand self-occlusion can cause non-physical transient changes in coordinate points in the raw data stream collected by the sensor, directly disrupting the logical continuity of the feature sequence.

[0003] Industry-standard improvement approaches often focus on increasing hardware redundancy or deploying large-scale neural network models. While these attempts can alleviate some feature recognition challenges, they often introduce new systemic contradictions. Improving hardware performance inevitably increases deployment costs, and high-load computational logic causes significant processing latency, failing to meet the low-latency requirements of precise interactions. Existing designs tend to treat interconnected joints as isolated independent variables, failing to fully utilize the inherent topological connectivity and biomechanical constraints of the hand skeleton. This results in a lack of logical-level verification and compensation capabilities for local data distortions. At the backend algorithm logic level, existing technologies have failed to eliminate coordinate redundancy and physical distortion constraints. For example, Chinese invention patent application CN113312966A discloses a first-person perspective action recognition method and device that uses an adaptive graph convolutional network (AGCN) to aggregate and recognize skeletal features. The core logic is anchored in visual space feature probability fitting. Such solutions lack physical modeling of the inherent rigid features of the hand skeleton, making it difficult to output logical deterministic verification feedback when faced with non-physical coordinate transients caused by occlusion. Furthermore, relying on deep neural network nonlinear mapping leads to cumulative perceptual offsets in low-latency interaction scenarios.

[0004] Therefore, how to construct a topological feature mapping mechanism that is scale-independent and has logical repair capabilities to eliminate the fragility of absolute spatial coordinate mapping and solve the recognition distortion and feedback delay in the interaction process has become the technical problem to be solved by this invention. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a gesture interaction system for multi-dimensional gesture command recognition, comprising: The coordinate acquisition module is used to acquire the original coordinate sequence of hand joints in three-dimensional space; The topology modeling module is used to create a skeletal topology map defined by hand joints based on the preset hand bone connection relationships. The skeletal topology map contains multiple topological edges connecting adjacent hand joints. The variation coefficient extraction module is used to calculate the rate of change of the side length of each topological edge over time based on the original coordinate sequence, so as to generate dynamic variation coefficients that characterize the skeletal geometric constraints. The jump arbitration module is used to compare the dynamic variation coefficient of each topological edge with the preset geometric consistency threshold. If the dynamic variation coefficient of a specific topological edge exceeds the geometric consistency threshold and the dynamic variation coefficient of the other topological edges is within the preset elastic variation range of 0.08 to 0.12, the hand joint connecting the specific topological edge is determined to be a non-physical jump point. The coordinate prediction module is used to perform logical prediction of non-physical jump points based on the spatial orientation of the remaining hand joints in the skeletal topology map, in order to fill in the feature blind spots in the original coordinate sequence. The instruction conversion module is used to convert the completed original coordinate sequence into gesture interaction control instructions through an interactive coordinate mapping matrix.

[0006] Preferably, when generating dynamic variation coefficients, the variation coefficient extraction module is also used to obtain the motion vector magnitude of the hand joints, and to linearly adjust the tolerance width of the geometric consistency threshold in real time based on the motion vector magnitude. The tolerance width is positively correlated with the motion vector magnitude, and is used to preload the matching parameters of the instruction conversion module by expanding the judgment window of the jump arbitration module under high-speed motion conditions.

[0007] Preferably, the topology modeling module maps the original coordinate sequence into a graph domain signal and uses the graph Laplacian operator to smooth and filter the graph domain signal to generate a skeletal topology signal after noise filtering. The variation coefficient extraction module calculates the dynamic variation coefficient based on the skeletal topology signal.

[0008] Preferably, when the coordinate prediction module completes the logical prediction, it retrieves the adjacent nodes connected to the non-physical jump point in the skeletal topology map, and inverts the predicted spatial coordinates of the non-physical jump point based on the displacement components of the adjacent nodes and the preset rigid constraint distance of the skeletal structure.

[0009] Preferably, the instruction conversion module includes a spatiotemporal convolution extraction unit, which is used to extract multi-frame temporal features from the completed original coordinate sequence, generate a semantic feature vector of the dynamic gesture, and perform cosine similarity matching between the semantic feature vector and a preset gesture instruction template library.

[0010] Preferably, when processing the original coordinate sequence, the system uses the coordinate prediction module to complete the in-situ coordinate restoration of the hand joints that are determined to be occluded or lost, and the instruction conversion module maintains the continuity of the gesture interaction path based on the in-situ coordinate restoration sequence.

[0011] Preferably, the coordinate acquisition module is connected to a depth image sensor, which is used to acquire three-dimensional coordinate values ​​containing depth information and encapsulate the three-dimensional coordinate values ​​into a raw coordinate sequence and input it into the topology modeling module.

[0012] Preferably, the topology modeling module is also used to scale the reference length of each topological edge in the skeletal topology map proportionally according to the palm size measurement values ​​of different users, so that the geometric consistency threshold generated by the jump arbitration module can be adapted to the heterogeneous skeletal anatomical features.

[0013] Preferably, before outputting the gesture interaction control command, the command conversion module extracts the overall geometric distribution features of the original coordinate sequence, and issues the gesture interaction control command when the relative angles of each hand joint are within a preset reasonable biomechanical value range.

[0014] The beneficial effects of this invention are: 1. In multidimensional gesture command recognition, by kinematic topological modeling of the connection relationship of hand joints, the system converts the collected original spatial coordinates into dynamic strain coefficients that characterize the rate of change of topological side lengths. Since the strain coefficients depend only on the relative deformation of the internal structure of the hand, they eliminate the interference of differences in the physical size of the user's hand and the relative displacement between the user and the sensor on feature extraction, so that the same gesture command template can be adapted to the heterogeneous physical characteristics of different individuals, and achieve scale independence of feature recognition.

[0015] 2. Based on the rigid constraint field established by the hand skeleton topology map, the system performs consistency audit on the strain coefficient of each topological edge in continuous frames. When a local joint point causes a non-physical jump due to environmental noise or hand self-occlusion, the abnormal change in the strain coefficient triggers the logic threshold check. The system performs in-situ logic compensation on the jump point according to the spatial orientation of the other stable joint points and the preset skeleton connection distance. This self-repair capability based on endogenous topological constraints maintains the physical consistency of the gesture feature flow and effectively avoids the risk of command mis-triggering caused by non-physical jumps.

[0016] 3. The dynamic window coefficient and motion trend prediction mechanism work together to enable the system to adjust the fault tolerance range of feature matching in real time according to the angular acceleration of the user's gesture. In high-speed gesture motion scenarios, the system preloads the synchronization parameters of the feedback object by identifying the evolution trend of the strain tensor, compensating for the perception delay caused by sampling distortion or data processing time. This cross-dimensional feature coupling processing scheme achieves deep synchronization of physical motion and virtual feedback at the logical level without increasing the sampling frequency of the sensor hardware. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a schematic diagram of the functional module integration principle of the gesture interaction system of the present invention; Figure 2 This is a flowchart of the non-physical jump point arbitration and coordinate prediction process of the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, an embodiment or embodiment referred to herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. An embodiment appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0021] This invention is described in detail with reference to the schematic diagrams. When describing the embodiments of this invention, for ease of explanation, the cross-sectional views of the device structure will be partially enlarged without adhering to the general scale. Moreover, the schematic diagrams are only examples and should not limit the scope of protection of this invention. In addition, in actual manufacturing, the three-dimensional spatial dimensions of length, width and depth should be included.

[0022] Furthermore, in the description of this invention, it should be noted that the terms such as "upper," "lower," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or component referred to has a specific orientation, or is constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] Unless otherwise explicitly specified and limited, the terms installation, connection, and linking in this invention should be interpreted broadly. For example, they can refer to fixed connection, detachable connection, or integrated connection; similarly, they can refer to mechanical connection, electrical connection, or direct connection, or indirect connection through an intermediate medium, or internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] A gesture interaction system for multidimensional gesture command recognition, comprising: The coordinate acquisition module is used to acquire the original coordinate sequence of hand joints in three-dimensional space; The topology modeling module is used to create a skeletal topology map defined by hand joints based on the preset hand bone connection relationships. The skeletal topology map contains multiple topological edges connecting adjacent hand joints. The variation coefficient extraction module is used to calculate the rate of change of the side length of each topological edge over time based on the original coordinate sequence, so as to generate dynamic variation coefficients that characterize the skeletal geometric constraints. The jump arbitration module is used to compare the dynamic variation coefficient of each topological edge with the preset geometric consistency threshold. If the dynamic variation coefficient of a specific topological edge exceeds the geometric consistency threshold and the dynamic variation coefficient of the other topological edges is within the preset elastic variation range of 0.08 to 0.12, the hand joint connecting the specific topological edge is determined to be a non-physical jump point. The coordinate prediction module is used to perform logical prediction of non-physical jump points based on the spatial orientation of the remaining hand joints in the skeletal topology map, in order to fill in the feature blind spots in the original coordinate sequence. The instruction conversion module is used to convert the completed original coordinate sequence into gesture interaction control instructions through an interactive coordinate mapping matrix.

[0025] Preferably, when generating dynamic variation coefficients, the variation coefficient extraction module is also used to obtain the motion vector magnitude of the hand joints, and to linearly adjust the tolerance width of the geometric consistency threshold in real time based on the motion vector magnitude. The tolerance width is positively correlated with the motion vector magnitude, and is used to preload the matching parameters of the instruction conversion module by expanding the judgment window of the jump arbitration module under high-speed motion conditions.

[0026] Preferably, the topology modeling module maps the original coordinate sequence into a graph domain signal and uses the graph Laplacian operator to smooth and filter the graph domain signal to generate a skeletal topology signal after noise filtering. The variation coefficient extraction module calculates the dynamic variation coefficient based on the skeletal topology signal.

[0027] Preferably, when the coordinate prediction module completes the logical prediction, it retrieves the adjacent nodes connected to the non-physical jump point in the skeletal topology map, and inverts the predicted spatial coordinates of the non-physical jump point based on the displacement components of the adjacent nodes and the preset rigid constraint distance of the skeletal structure.

[0028] Preferably, the instruction conversion module includes a spatiotemporal convolution extraction unit, which is used to extract multi-frame temporal features from the completed original coordinate sequence, generate a semantic feature vector of the dynamic gesture, and perform cosine similarity matching between the semantic feature vector and a preset gesture instruction template library.

[0029] Preferably, when processing the original coordinate sequence, the system uses the coordinate prediction module to complete the in-situ coordinate restoration of the hand joints that are determined to be occluded or lost, and the instruction conversion module maintains the continuity of the gesture interaction path based on the in-situ coordinate restoration sequence.

[0030] Preferably, the coordinate acquisition module is connected to a depth image sensor, which is used to acquire three-dimensional coordinate values ​​containing depth information and encapsulate the three-dimensional coordinate values ​​into a raw coordinate sequence and input it into the topology modeling module.

[0031] Preferably, the topology modeling module is also used to scale the reference length of each topological edge in the skeletal topology map proportionally according to the palm size measurement values ​​of different users, so that the geometric consistency threshold generated by the jump arbitration module can be adapted to the heterogeneous skeletal anatomical features.

[0032] Preferably, before outputting the gesture interaction control command, the command conversion module extracts the overall geometric distribution features of the original coordinate sequence, and issues the gesture interaction control command when the relative angles of each hand joint are within a preset reasonable biomechanical value range.

[0033] Example 1: In the application of virtual assembly of electronic components, operators wearing protective equipment execute gesture commands in an environment with dynamic interference. The gesture interaction system collects hand gestures transmitted by sensors in real time. The original coordinate sequence of each joint point changes with the shooting distance and anatomical structure when the operator moves at the assembly station or when there are differences in individual hand size. Furthermore, self-occlusion of the hand during assembly causes non-physical jumps in the original joint point coordinates, deviating from biomechanical constraints, leading to breaks in the interaction path. The gesture interaction system uses a topology modeling module to establish a skeletal topology map composed of hand joint points based on preset hand bone connection relationships. The variation coefficient extraction module calculates the rate of change of the edge length over time for each topological edge in the skeletal topology map, generating dynamic variation coefficients that characterize the geometric constraints of the skeleton. subscript and These represent the key point indexes.

[0034] The extraction of the topological side length change rate transforms absolute spatial coordinates into scalar features characterizing the relative deformation of the skeleton, eliminating the influence of user physical dimensions and shooting distance on feature extraction. The jump arbitration module then uses the dynamic variation coefficient. Compared with a preset geometric consistency threshold, if the dynamic variation coefficient of a specific topological edge exceeds the threshold and the dynamic variation coefficient of the remaining topological edges is within a preset range of 0.08 to 0.12, the connected joint is determined to be a non-physical jump point. The coordinate prediction module retrieves the adjacent nodes connected to the non-physical jump point in the skeletal topology map, and completes the coordinate prediction based on the displacement components of the adjacent nodes and the preset skeletal rigidity constraint distance to fill in the feature blind spots in the original coordinate sequence. The instruction conversion module converts the completed original coordinate sequence into gesture interaction control instructions through the interactive coordinate mapping matrix. The system provides logical verification for the data stream through the physical rigidity constraint of the skeletal structure, decouples the interaction accuracy from the hardware sampling frequency, and realizes stable feedback and instruction output under working conditions.

[0035] Example 2: To verify the tracking performance of the gesture interaction system in the virtual assembly task of electronic components, an image sensor with a sampling frequency of 30Hz and a spatial resolution better than 1mm was used to output the original coordinate sequence of 21 hand joints. The test environment simulated the dynamic lighting conditions of a precision assembly workshop, and Gaussian white noise with a signal-to-noise ratio of 20dB was superimposed on the signal source to characterize the electromagnetic interference in the industrial environment. This was used to construct a physical experimental platform for obtaining the coordinates of the hand joints. The experiment involved setting a geometric consistency threshold. The established procedure is based on a logic that balances the sensitivity of capturing real human movements with the strength of suppressing coordinate jumps caused by hand self-occlusion. By collecting 500 frames of hand motion data in different dimensions under unobstructed standard conditions, it was found that the rate of change of topological side length induced by normal physiological displacement mainly ranges from 0.03 to 0.06. To allow for fluctuations in sensor acquisition accuracy and elastic deformation of skin tissue, a geometric consistency threshold was set. The value is set to 0.10, and a flexible variation range of 0.08 to 0.12 is established accordingly as the basis for the jump arbitration module to execute the logic judgment. As a dimensionless threshold, in the verification process for scale-independent feature extraction, a first test sample group with a palm length of 165mm and a second test sample group with a palm length of 192mm were selected. Both groups executed the same interactive commands at gradient distances of 0.5m, 1.0m, and 1.5m from the sensor. In the control group using absolute spatial coordinate mapping, the recognition error rate of the first test sample group at 1.5m increased to 18.4% as the projection scale decreased. However, in the test using the sample group of this invention, the variation coefficient extraction module converted the original coordinate sequence into dynamic variation coefficients characterizing structural deformation. The experimental results show that the feature response curves of the first test sample group and the second test sample group tend to be consistent at different shooting distances, and their average recognition deviations converge to 1.8% and 2.1% respectively, proving that the topological side length change rate eliminates the nonlinear shift of feature vectors caused by physical size differences and spatial displacement.

[0036] In the experiment targeting non-physical jump suppression capabilities, a mechanical baffle was used to randomly block the visual path of fingertip joints during the assembly process. In the original coordinate sequence, abrupt displacement changes were observed at the blocked joints due to signal loss, with the corresponding topological side length change rate instantaneously exceeding 0.18. At this point, the control group, lacking verification of skeletal rigidity constraints, interpreted the data abrupt change as hand gesture displacement and generated virtual image jitter at a frequency of 5Hz. However, the jump arbitration module of this invention detected the topological side length change rate exceeding the geometric consistency threshold. When a point is determined to be a non-physical jump point, the coordinate prediction module extracts the displacement components of adjacent nodes connected to that point in the skeletal topology map and completes the coordinates based on the preset skeletal rigidity constraint distance. Data analysis shows that when there are five consecutive frames of local occlusion, the output deviation of the interactive coordinate mapping matrix of the control group reaches 42.5 mm, while the coordinate deviation of the sample group of this invention after logical prediction remains within 3.2 mm. The continuity of system command response is improved from 76.4% in the control group to 98.5%. This proves that the system uses the biomechanical invariance of the hand bone connection to provide a physically self-consistent verification mechanism for the data flow. Without changing the sampling frequency of the sensor hardware, the system achieves robustness of command recognition in complex industrial environments through nonlinear judgment and compensation of topological strain characteristics.

[0037] Example 3: This example combines Figures 1 to 2 A description of a gesture interaction system used for multidimensional gesture command recognition, such as... Figure 1As shown, the multi-dimensional gesture command recognition gesture interaction system includes a coordinate acquisition module, a topology modeling module, a variation coefficient extraction module, a jump arbitration module, a coordinate prediction module, and a command conversion module. The coordinate acquisition module acquires the original coordinate sequence and transmits it to the topology modeling module, the variation coefficient extraction module, and the coordinate prediction module respectively. The topology modeling module builds a skeletal topology map based on the original coordinate sequence and inputs it to the variation coefficient extraction module and the coordinate prediction module respectively. The variation coefficient extraction module combines the original coordinate sequence and the skeletal topology map to generate dynamic variation coefficients and outputs them to the jump arbitration module. The jump arbitration module identifies non-physical jump points through judgment logic and transmits them to the coordinate prediction module. The coordinate prediction module combines the original coordinate sequence, the skeletal topology map, and the non-physical jump points to complete logical prediction and outputs the completed original coordinate sequence to the command conversion module. Finally, the command conversion module converts the completed sequence into gesture interaction control commands.

[0038] like Figure 2 As shown, when the system performs jump determination and coordinate prediction, it obtains the dynamic variation coefficient and compares it with the preset determination criteria. If the dynamic variation coefficient is outside the elastic variation range of 0.08 to 0.12 and exceeds the threshold, the relevant joint point is determined to be a non-physical jump point. For the determined non-physical jump point, the system obtains its motion state information by searching the adjacent nodes, and performs the displacement component inversion of the adjacent nodes in combination with the preset bone rigidity constraint distance, thereby calculating and outputting the predicted spatial coordinates of the joint point.

[0039] Example 4: During system initialization, the topology modeling module acquires the original coordinate sequence of 21 hand joints transmitted by the coordinate acquisition module, and pre-assigns each joint to node 0 through 20 according to biological anatomy. Node 0 represents the center of the wrist, nodes 1 through 4 correspond to the base of the thumb to the fingertip, and so on. The module establishes topological edges between adjacent joints with physiological connections based on a pre-defined adjacency matrix. The resulting skeletal topology map contains 20 lines representing bone length, thus establishing the rigid constraint boundary of the hand structure. The variation coefficient extraction module captures two consecutive frames of images during system operation to obtain node... With nodes exist Spatial coordinates of time and The topological side length at that moment is obtained by calculating the Euclidean distance between the two. According to the formula Determine the dynamic variation coefficient, where... The sampling time interval of the sensor. To characterize the scalar features of the relative deformation rate of the skeleton, this procedure converts the absolute displacement of the original coordinates into the endogenous rate of change of the topology, eliminating the interference of individual size differences of operators on instruction extraction.

[0040] The elastic variation range of 0.08 to 0.12 used in the jump arbitration module is derived from statistical calibration of 1000 sets of dynamic gesture samples. Actual measurement data shows that the pseudo-deformation rate caused by the topological side length displacement of human finger bones under normal physiological movements is usually below 0.05. However, when occlusion interference causes positioning inaccuracies, the calculated dynamic variation coefficient... It will exceed physiological limits and jump to above 0.20; therefore, the system will set the geometric consistency threshold. Anchored to 0.10 and reserving 20% ​​fault-tolerant bandwidth, the jump arbitration module ensures that it only determines non-physical jump points when it detects data offsets that violate physical rigid constraints. During the jump arbitration, the variation coefficient extraction module, based on the physical constraint relationship that the uncertainty of rigid body displacement increases linearly with instantaneous velocity in kinematics, executes the dynamic adjustment procedure of the geometric consistency threshold in real time. The system is set up to include a multi-core processing unit computing environment to extract the spatial coordinate difference between two consecutive frames of the hand's topological centroid in each sampling period, calculate the motion vector magnitude, and the processing unit determines the dynamic threshold of the current frame based on the logic that the dynamic threshold equals the reference parameter plus the product of the adjustment coefficient and the motion vector magnitude. The processing unit is based on the formula... Determine the dynamic threshold for the current frame, where For dynamic thresholds, As the baseline parameter, For adjustment coefficients, The motion vector magnitude is defined as follows: the dynamic threshold is a geometrically consistent dimensionless threshold adjusted in real time with legal value boundaries limited to 0.10 to 0.25; the baseline parameter is a geometrically consistent baseline parameter with a value of 0.10 under static conditions; the standard unit for the extracted numerical value of the motion vector magnitude is meters per second; and the adjustment coefficient is a linear adjustment coefficient representing the system's following sensitivity, with units of seconds per meter. The system records the mapping relationship between the motion vector and the abrupt boundary of the variation coefficient under standard unobstructed conditions, calibrating the adjustment coefficient to 0.05. The linear adjustment calculation logic is executed. The system synchronously expands the jump judgment window as the operator accelerates with their gestures. During the synchronous expansion of the judgment window, the processing unit extracts the gradient rate of change of the current motion vector magnitude and converts it into a pre-defined value. The alarm signal is sent to the instruction conversion module. After receiving the synchronous alarm signal, the instruction conversion module loads the set of gesture instruction templates with the highest correlation to the current direction of motion into the high-speed computing cache queue in batches from the system memory. At the same time, it proportionally lowers the start trigger benchmark value of the cosine similarity matching logic to complete the preloading operation of the matching parameters. This directly compensates for the perception delay caused by real-time data comparison in high-dynamic computing scenarios and eliminates high-frequency random fluctuations during spatial coordinate sampling. The topology modeling module performs a spatial smoothing filtering procedure on the original coordinate sequence based on the general law of high correlation between the physical states of connected nodes in graph signal processing. The processing unit establishes an adjacency matrix with a preset 21x21 dimension skeletal connection relationship. When the first node is connected to the second node, the adjacency matrix is ​​established. When two nodes have a direct anatomical connection, the corresponding matrix element is assigned the inverse exponential decay term of the spatial distance, and the remaining elements are assigned 0. The processing unit uses the established adjacency matrix to calculate the standardized graph Laplacian operator applied to the original coordinate sequence, extracting the skeletal topology signal after low-pass filtering. In the specific calculation and construction steps, the processing unit obtains the corresponding degree matrix based on the established adjacency matrix, extracts the inverse square root of the degree matrix, multiplies it with the adjacency matrix, and then subtracts it from the identity matrix to calculate the standardized graph Laplacian matrix system. A low-pass graph filter is constructed using the eigenvectors corresponding to a specific number of minimum eigenvalues ​​extracted from this graph Laplacian matrix. By applying the original coordinate sequence containing high-frequency components of spatial jumps to the low-pass filter... The graph filter performs spatial multiplication operations to achieve spectral aggregation and dimensionality reduction of the feature dimension, completing spectral smoothing and noise reduction. Before outputting control commands, the instruction conversion module performs a compliance verification procedure for the overall geometric distribution features based on the mechanical rigidity limit of human joint rotation angles and biomechanical mechanisms. The processing unit calculates the relative angles between each phalanx using the cosine theorem of the spatial vector angle based on the completed three-dimensional spatial coordinates as a quantitative index of the overall geometric distribution features. The system sets up an array containing multiple numerical boundary verifications to limit the reasonable relative angle range of adjacent phalanges to 0 degrees to 135 degrees. If any relative angle calculated exceeds the preset range, the system determines that the current feature is incorrect and reconstructs and intercepts the signal in the specific range; otherwise, the feature flow is maintained and passed down.

[0041] After receiving the completed original coordinate sequence, the instruction conversion module calls the interactive coordinate mapping matrix. Perform a spatial projection transformation. This matrix aligns the local coordinate system containing the hand's topological centroid to the global coordinate system of the virtual display device using preset translation and rotation operators. The transformation process follows... The linear mapping logic, where, Given an input vector containing the completed coordinates, This is an interactive coordinate mapping matrix that includes scaling factors and offsets. To generate the final control command coordinates, this procedure transforms the microscopic hand deformation features into a macroscopic interactive command flow, maintaining real-time synchronization between physical motion and digital feedback. Based on this spatial image, the spatiotemporal convolution extraction unit, which is embedded within the command conversion module, performs a multi-frame temporal feature extraction process on the continuous feature sequence after projection transformation. This extraction unit includes a spatial graph convolutional layer and a one-dimensional temporal convolutional layer processing unit. Using two convolutional structures, the unit alternately fuses and reduces the dimensionality of the node spatial position distribution within a single frame and the node motion trajectory features between consecutive frames. After the high-dimensional coordinate sequence is processed by the hierarchical alternating convolution and global average pooling layer feature compression operations, it is accurately transformed into a fixed 512-dimensional semantic feature vector containing spatiotemporal information of the action. The system constructs a self-consistent verification procedure at the bottom layer of data processing by defining the rigid topology of the skeletal connection relationship and quantifying the coefficient of variation. This procedure uses the inherent rigidity of biological structures to achieve logical filtering of environmental noise, reducing the false trigger rate of command recognition.

[0042] Example 5: In a field deployment scenario of a gesture interaction system, the system initiates a spatial alignment procedure to determine the interaction coordinate mapping matrix. The internal transformation parameters are used by the coordinate acquisition module to obtain the three-dimensional spatial coordinates of multiple reference points within the effective field of view of the sensor. The topology modeling module synchronously calculates the local coordinates of the topological centroid of the gesture feature point set. The processor calculates the rotation operator that minimizes the spatial residual function based on the least squares decision logic. With translation operator By using the rotation operator With translation operator The matrix parameters are initialized by writing to memory. This procedure establishes a linear mapping path between the hand topology space and the global coordinate system of the virtual display device.

[0043] When the system encounters sensor hardware with different signal-to-noise ratio specifications or operating conditions with environmental photoelectric interference, the jump arbitration module initiates a pre-calibration process to dynamically determine the geometric consistency threshold. The variation coefficient extraction module collects data during the initial static period after system startup. The original coordinate sequence of the frame is obtained, and the background variation rate caused by environmental noise is calculated for each topological edge in the skeletal topology map. The statistical mean of the processor's statistical baseline change rate with standard deviation And according to the formula The decision threshold is modified, among which... To determine the threshold, This is the statistical mean of the rate of change of the background. As the standard deviation of the baseline rate of change, this process provides a logical benchmark for subsequent jump arbitration that is adapted to the current deployment environment, maintaining the system's identification stability when applied across hardware platforms.

[0044] In industrial monitoring scenarios where the gesture interaction system is running continuously, the system initiates an adaptive drift compensation procedure to maintain the interaction coordinate mapping matrix. To ensure stability, the processor monitors the spatial distribution of the hand's topological centroid through a sliding time window of 300 frames and calculates the cumulative variance of the output coordinates within the sliding time window. If cumulative variance is detected If the trend term shows a sustained unidirectional shift and the deviation exceeds 15% of the initial baseline value within a preset 60 seconds, it is determined that the coordinate axis is misaligned due to slight changes in the physical position of the sensor or drift in ambient temperature. At this time, the system triggers a background reconstruction command, and without interrupting the current interactive process, it reconstructs the coordinates based on the latest three-dimensional spatial coordinates within the sliding window. Local coordinates of the topological centroid Recalculate the rotation operator With translation operator Furthermore, by using a weighted fusion algorithm, the original matrix parameters are smoothly transitioned to the newly generated parameter set. This procedure utilizes dynamic data feedback to correct geometric deviations caused by environmental changes, ensuring the coordinate mapping accuracy of the system under long-term operating conditions.

[0045] Example 6: In the parameter calibration scenario of a gesture interaction system, the system sets a geometric consistency threshold for the transition arbitration module. The standardized calibration procedure is initiated. During the initial quasi-static period after system startup, the variation coefficient extraction module acquires 100 frames of raw coordinate sequences. The calculation unit calculates the background variation rate caused by thermal noise of the acquisition system for each topological edge in the skeletal topology map. The processor then calculates the statistical mean of the background variation rate. with standard deviation And according to the formula The procedure modifies the decision threshold, providing a logical baseline for subsequent jump arbitration that is adapted to the current deployment environment. To determine the threshold, This is the statistical mean of the rate of change of the background. The standard deviation of the background rate of change.

[0046] When the system determines that a non-physical jump point has occurred in the original coordinate sequence, the coordinate prediction module extracts the reference bone length of the corresponding topological edge pre-stored in the memory. The motion trend direction vector is calculated using the coordinates of adjacent nodes at non-physical jump points. The completed three-dimensional spatial coordinates are obtained through the formula It is confirmed that, among them, To predict the generated 3D coordinates of the joints, The real-time three-dimensional spatial coordinates of adjacent nodes. Based on the baseline bone length, To obtain the direction unit vector by calculating the displacement vector of adjacent nodes between the current time and the previous sampling time and performing normalization processing, this procedure utilizes the length invariance of the hand skeletal structure to lock the prediction result within the physiological motion trajectory envelope. The command coordinate deviation output by the command conversion module converges to within 3mm. In the above inversion calculation stage, considering the objective spatial angle deviation between the translation direction of the bone end and the extension direction of the bone in kinematics, the system further multiplies the extracted direction unit vector by a preset finger joint rotation mapping tensor. This mapping tensor is dynamically updated and extracted based on the relative positional relationship of multiple stable skeletal nodes in three adjacent frames, thereby accurately correcting the translation velocity vector of adjacent nodes to the pointing vector along the direction of the bone topology connection, ensuring that the spatial prediction mechanism is completely consistent with the common sense of rigid body physics.

[0047] When the system encounters a situation where an operator performs highly dynamic hand gestures, the coordinate prediction module initiates an alternative implementation procedure based on second-order dynamic constraints. The variation coefficient extraction module detects that the instantaneous displacement velocity of a specific node exceeds a preset velocity threshold. At this point, linear interpolation compensation is stopped, and instead, the pre-stored hand skeletal motion inertial parameters are called. Specifically, the coordinate prediction module uses the formula... Determine the compensation coordinates, where, The generated joint point three-dimensional spatial coordinates, These are the three-dimensional spatial coordinates of the joints at the previous sampling time. The instantaneous velocity vector extracted from the previous frame. The acceleration vector is calculated based on the coordinates of three consecutive frames. The sampling time interval is used to solve the problem of prediction failure when acceleration changes suddenly by utilizing the physical inertia consistency of biological motion, so that the system can maintain the logical closed loop of feature flow in dynamic interactive scenarios.

[0048] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A gesture interaction system for multi-dimensional gesture command recognition, characterized in that, include: The coordinate acquisition module is used to acquire the original coordinate sequence of hand joints in three-dimensional space; The topology modeling module is used to create a skeletal topology map defined by hand joints based on the preset hand bone connection relationships. The skeletal topology map contains multiple topological edges connecting adjacent hand joints. The variation coefficient extraction module is used to calculate the rate of change of the side length of each topological edge over time based on the original coordinate sequence, so as to generate dynamic variation coefficients that characterize the skeletal geometric constraints. The jump arbitration module is used to compare the dynamic variation coefficient of each topological edge with the preset geometric consistency threshold. If the dynamic variation coefficient of a specific topological edge exceeds the geometric consistency threshold and the dynamic variation coefficient of the other topological edges is within the preset elastic variation range of 0.08 to 0.12, the hand joint connecting the specific topological edge is determined to be a non-physical jump point. The coordinate prediction module is used to perform logical prediction of non-physical jump points based on the spatial orientation of the remaining hand joints in the skeletal topology map, in order to fill in the feature blind spots in the original coordinate sequence. The instruction conversion module is used to convert the completed original coordinate sequence into gesture interaction control instructions through an interactive coordinate mapping matrix.

2. The gesture interaction system for multi-dimensional gesture command recognition according to claim 1, characterized in that, When generating dynamic variation coefficients, the variation coefficient extraction module is also used to obtain the motion vector magnitude of the hand joints, and adjusts the tolerance width of the geometric consistency threshold linearly in real time based on the motion vector magnitude. The tolerance width is positively correlated with the motion vector magnitude, and is used to preload the matching parameters of the instruction conversion module by expanding the decision window of the jump arbitration module under high-speed motion conditions.

3. A gesture interaction system for multi-dimensional gesture command recognition according to claim 1, characterized in that, The topology modeling module maps the original coordinate sequence into a graph domain signal and uses the graph Laplacian operator to smooth and filter the graph domain signal to generate a skeletal topology signal after noise filtering. The variation coefficient extraction module calculates the dynamic variation coefficient based on the skeletal topology signal.

4. A gesture interaction system for multi-dimensional gesture command recognition according to claim 1, characterized in that, When performing logical prediction, the coordinate prediction module retrieves adjacent nodes in the skeletal topology map that are connected to non-physical jump points, and inverts the predicted spatial coordinates of non-physical jump points based on the displacement components of the adjacent nodes and the preset rigid constraint distance of the skeletal structure.

5. A gesture interaction system for multi-dimensional gesture command recognition according to claim 1, characterized in that, The instruction conversion module includes a spatiotemporal convolution extraction unit, which is used to extract multi-frame temporal features from the completed original coordinate sequence, generate a semantic feature vector of the dynamic gesture, and perform cosine similarity matching between the semantic feature vector and the preset gesture instruction template library.

6. A gesture interaction system for multi-dimensional gesture command recognition according to claim 1, characterized in that, When processing the original coordinate sequence, the system uses the coordinate prediction module to restore the in-situ coordinates of hand joints that are determined to be occluded or missing. The instruction conversion module maintains the continuity of the gesture interaction path based on the in-situ coordinate-restored sequence.

7. A gesture interaction system for multi-dimensional gesture command recognition according to claim 1, characterized in that, The coordinate acquisition module is connected to a depth image sensor, which is used to acquire three-dimensional coordinate values ​​containing depth information and encapsulates the three-dimensional coordinate values ​​into a raw coordinate sequence and inputs it into the topology modeling module.

8. A gesture interaction system for multi-dimensional gesture command recognition according to claim 1, characterized in that, The topology modeling module is also used to scale the reference length of each topological edge in the skeletal topology map proportionally according to the palm size measurements of different users, so that the geometric consistency threshold generated by the jump arbitration module can be adapted to heterogeneous skeletal anatomical features.

9. A gesture interaction system for multi-dimensional gesture command recognition according to claim 1, characterized in that, Before outputting gesture interaction control commands, the command conversion module extracts the overall geometric distribution features of the original coordinate sequence and issues gesture interaction control commands when the relative angles of each hand joint are within the preset reasonable biomechanical value range.

Images