A visual multi-modal non-contact gesture unlocking method

Abstract

The application discloses a visual multi-modal non-contact gesture unlocking method, which comprises the following steps: S1, collecting multi-source sensing data; S2, pre-processing RGB image data; S3, fusing event flow and RGB image, taking the features of fingers and palms as vertices and connecting lines as edges to express the spatial structure of hands; the time correlation is constructed by connecting lines of the same vertices in the front and rear continuous frames as edges, and then the event flow is dynamically mapped into a graph structure to obtain multi-modal fusion features; S4, multi-modal joint prediction, mapping the multi-modal fusion features into a neural network model of a latent space, outputting parameters mu and sigma of a latent distribution, and generating latent features Z through a reparameterization technique; after the latent features Z are superimposed with an additional condition channel, the superimposed features are decoded through a multi-layer perception (MLP) to obtain a final hand posture prediction. The application provides a low-delay, high-robustness real-time gesture interaction and high-precision sign language recognition suitable for challenging environments such as light changes and rapid motions.

Description

Technical Field

[0001] This invention belongs to the field of gesture tracking and recognition technology, specifically relating to a visual multimodal non-contact gesture unlocking method. Background Technology

[0002] With the rapid development of technology, people's demands for safety and convenience in life are constantly increasing, and the importance of public health and safety is becoming increasingly prominent. Traditional mechanical locks and contact-based electronic locks are proving inadequate in meeting modern high standards of security and hygiene, driving the rapid development of contactless unlocking technologies. Existing contactless unlocking technologies mainly include Radio Frequency Identification (RFID), Near Field Communication (NFC), biometrics (such as iris recognition and facial recognition), and gesture recognition based on machine vision. Although these technologies have been widely used in areas such as smart locks, mobile phone unlocking, and electronic payments, they still have many limitations:

[0003] Radio Frequency Identification (RFID) and Near Field Communication (NFC) are susceptible to electromagnetic interference, resulting in unstable unlocking performance and low reliability in complex environments.

[0004] Biometric technology: Although it has high security, there is still room for improvement in terms of recognition speed and system modifiability, and recognition performance is limited under certain lighting conditions.

[0005] Machine vision-based gesture recognition: Traditional cameras perform poorly due to high power consumption, high latency, and low dynamic range, limiting their application in real-time and dynamic environments. Furthermore, existing vision systems struggle to balance recognition accuracy and response speed in complex backgrounds or with rapid gesture changes.

[0006] Single-modal recognition, such as RGB images, suffers from insufficient accuracy under complex lighting and background interference. Its limited dynamic range leads to loss of detail, high computational complexity makes it difficult to meet real-time requirements, lacks effective capture of temporal correlation, and is affected by background noise interference. Furthermore, the key generation and transmission process has low security, and there is a lack of effective user feedback mechanisms when verification fails. These issues limit the accuracy, security, real-time performance, and user experience of gesture recognition technology.

[0007] Therefore, how to solve the problem of synchronously acquiring high dynamic range and rich texture information, ensure that key features are effectively weighted during the sampling process to highlight hand features and suppress background noise, build a system architecture that can accurately capture and track the user's continuous action sequence, and convert this data into text form and encrypt and save it for verification are technical problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0008] The purpose of this invention is to provide a visual multimodal contactless gesture unlocking method to address the problems in the prior art.

[0009] Therefore, the above-mentioned objectives of the present invention are achieved through the following technical solutions:

[0010] A visual multimodal contactless gesture unlocking method, characterized by the following steps:

[0011] S1. Acquisition of multi-source sensor data: The infrared ranging sensor module is used to detect the approaching hand movement, triggering and scheduling the multi-source camera module to acquire hand data, obtaining hand movement data with high dynamic range and rich texture information.

[0012] The preprocessing of S2.RGB image data involves determining the sampling points of the RGB frame position channel and color channel through the offset matrix, dynamically adjusting the position of each sampling point, and reweighting the sampling area to amplify the channels that express hand features more strongly, thus initially extracting hand features.

[0013] S3. The fusion of event stream and RGB image: Based on the initial extraction of hand features, the features of fingers and palm are used as vertices and the connecting lines are used as edges to express the spatial structure of the hand; the temporal correlation is constructed by connecting the same vertices in consecutive frames as edges, and then the event stream is dynamically mapped to the graph structure to obtain the multimodal fusion features.

[0014] S4. Multimodal joint prediction maps multimodal fusion features to a neural network model in the latent space, outputs parameters μ and σ of the latent distribution, and generates latent features Z through reparameterization techniques; the latent features Z are superimposed with additional conditional channels and then decoded by a multilayer perceptron (MLP) to obtain the final hand pose prediction.

[0015] In step S3, the joints extracted by the pre-trained hand pose estimation model are used as vertices. Multi-dimensional attributes such as shape, position, and motion state are fused, and temporal associations are constructed by dynamically connecting consecutive frames through an event stream. A hierarchical graph structure is used to explicitly model the gesture evolution process. Edge weights are adaptively adjusted using event stream timestamps. Through dynamic spatial topology weighting and timestamp weighting mechanisms, both hand structure representation and motion trend capture are optimized. Combined with a bi-branch graph convolutional network, the static features of the RGB image and the high-precision dynamic information of the event stream are deeply fused. While employing the above technical solutions, this invention can also employ or combine the following technical solutions:

[0016] As a preferred technical solution of the present invention: In step S1, during data acquisition, a portion of gesture data is captured as real data using multiple RGB-D cameras with multiple perspectives and an event camera. The RGB image is first adaptively interpolated and then a synthesized event stream is generated through an event simulator as synthesized data, and the true value is constructed.

[0017] As a preferred technical solution of the present invention, step S2 specifically includes the following steps:

[0018] S21: A lightweight neural network predicts the sampling offset for each pixel position, generating an offset tensor of shape ijk2. The offset dynamically adjusts the sampling position of the convolution kernel to focus on the key hand region. Bilinear interpolation is used to calculate the pixel value at the offset point. Regular convolution is then performed on the offset sampling region to obtain feature one.

[0019] S22, split feature 1 by channel, calculate the global average value of each channel, generate the weight of each channel through the activation function, and reweight the channels of the original feature map using learnable parameters to enhance the channels that express hand features, suppress background noise, and obtain the preprocessed hand feature map.

[0020] As a preferred technical solution of the present invention, step S3 specifically includes the following steps:

[0021] Step S3 specifically includes the following steps:

[0022] S31, Construct the vertices of the graph, taking key hand feature regions such as fingertips, palms, and joints as vertices of the graph, and introducing the output of the pre-trained model as vertex attributes to form a multi-dimensional vertex representation;

[0023] S32, construct the time layer of the graph, take consecutive frames in time as the graph layer, each frame corresponds to a hand graph structure, and the time layers are dynamically connected through event flow to realize the transmission of time sequence information;

[0024] S33: Construct spatial structure connections. In the same time frame, connect the vertices of hand features as edges to construct a spatial structure, and dynamically adjust the weights of the edges according to the spatial topological relationship of the gesture to enhance the expressive power of the spatial structure.

[0025] S34, Constructing Temporal Correlation: In adjacent frames, vertices at the same position are connected as edges to construct temporal correlation, and the temporal correlation is weighted by the timestamp information of the event stream to improve the modeling accuracy of time series.

[0026] S35, multimodal fusion and graph convolution, synchronizes the timestamps of the event stream with the RGB frames, generates a time window that matches the RGB frames by accumulating the positive and negative dual channels of the event stream, dynamically maps the time window to the graph structure stored in the adjacency matrix, performs graph convolution operation on the graph structure, and extracts multi-channel features of multimodal fusion. Each channel contains temporal, spatial and motion mode information.

[0027] As a preferred technical solution of the present invention: The pre-trained model is a deep learning framework that integrates a convolutional neural network (CNN) and a pose regression network. Through end-to-end training on a large-scale publicly available hand keypoint dataset, high-precision regression and recognition of the three-dimensional spatial coordinates and class attributes of hand keypoints are achieved. Specifically, the keypoints extracted by the pre-trained model from the input RGB image and event stream image are used as the vertices of the graph structure. Each vertex is simultaneously assigned multi-dimensional attributes, including: its position vector in three-dimensional space, the shape feature descriptor of the current frame, and the motion state encoding based on the model output. Further, the historical trajectory point features of the vertex in the recent time series can be extended and added to form a multi-dimensional spatio-temporal and semantic fusion vertex representation, realizing multi-angle perception and expression of the hand state. Among them, the pose regression network is the HRNet or MediaPipe Hands hand pose estimation model.

[0028] As a preferred technical solution of the present invention: The specific calculation methods for each dimension in the multi-dimensional attributes of the vertex are as follows: The shape attribute is calculated by performing binary segmentation on the hand region of each frame, combining the Euclidean distance, angle, and regional geometric features between keypoints, and further using principal component analysis or autoencoder for dimensionality reduction to generate a compact shape feature vector; The motion state is analyzed through the time difference and velocity vector of the vertex spatial position in consecutive frames to automatically distinguish motion modes such as stationary, translational, rotational, opening and closing, etc. The features can be described by, for example, Δx, Δy, Δz, and direction cosine; The pre-trained gesture classification attribute directly calls the gesture recognition model to classify the global gesture type of the current frame or time series segment, and attaches the category probability distribution vector to the vertex attribute; The above multi-dimensional attributes are finally concatenated into a high-dimensional feature vector to comprehensively represent the shape, motion, and category features of the vertex.

[0029] As a preferred technical solution of the present invention: The specific synchronization method for the event flow to connect consecutive frames is as follows: First, through the hardware time synchronization mechanism, the time bases of the event camera and the RGB-D camera acquisition module are unified to ensure that the time stamps of the event stream are strictly aligned with the acquisition time points of each RGB frame; Subsequently, within each RGB frame acquisition window, all events in the event stream that overlap with the window time are intercepted and divided into two channels, positive and negative, according to the event time stamps, and cumulative statistics are performed respectively to generate the positive and negative event count maps corresponding to the RGB frame time window. The algorithm process is as follows: For each frame [t, t+Δt], count all events in the event stream E={e i =(x i , y i , t i , p i )} that satisfy t≤t i <t+Δt, and accumulate p i =+1 and p i=-1, the number of two polarities, forming a shape like W. + (x, y), W - The positive and negative event window mapping of (x, y) is further used to construct the graph structure and fuse multimodal features, so as to achieve accurate spatial and temporal alignment and information enhancement of the event stream and RGB frames.

[0030] As a preferred technical solution of the present invention: the specific method of adaptively adjusting edge weights based on event stream timestamps is as follows: when constructing the graph structure, for each pair of vertices v i With v j The spatial weight w of the edge between them s The spatial distance d between vertices ij Calculated using the Gaussian kernel function, w s =exp(-d ij 2 / 2σ s 2 ), where σ s Spatial scale parameter; time weight w t Based on the event stream timest interval Δt ij Adaptive adjustment, employing a weight allocation method similar to time-decay, w t =exp(-Δt ij / τ), where τ is the time scale hyperparameter; the total weight W of the edge. ij =α·w s + β·w t α and β are adjustable weighting coefficients that can be adjusted according to the importance of spatial or temporal correlations. Spatial variable weights automatically increase / decrease spatial weights by detecting changes in vertex spatial topology in real time, while temporal variable weights dynamically fine-tune temporal weights by combining the burst intensity of event flow or the trend of local temporal changes, thereby improving the model's sensitivity to dramatic changes in spatial structure and dynamic evolution over time.

[0031] As a preferred technical solution of the present invention: the specific structure for performing graph convolution operation on the graph structure is to adopt a dual-branch dynamic graph convolutional network to process the adjacency matrix and node features from RGB static features and event stream dynamic features respectively; each branch contains multiple layers of spatial graph convolution, realizing cross-channel propagation of spatial and temporal information in each layer, and introducing feature normalization and nonlinear activation after each layer; the multimodal feature fusion mechanism is an interactive attention mechanism and feature concatenation in parallel, first performing attention-weighted fusion of the two modal features, then concatenating them in the fusion layer and passing them to the downstream MLP decoder; the process of mapping the time window to the adjacency matrix is ​​as follows: using the positive and negative event graphs of the event stream accumulated in each time window as weights, the corresponding element A in the adjacency matrix A(t) is updated in real time. ij(t), that is, if the intensity of an event within the window is greater than the threshold, the weight of the relevant edge is increased, and vice versa; this process is automatically triggered by a gating mechanism or threshold discrimination, so as to realize the dynamic evolution of the adjacency matrix in time and the injection of multimodal information, ensuring that the graph structure can continuously and adaptively reflect the changes in hand state and action.

[0032] As a preferred technical solution of the present invention, step S4 specifically includes the following steps:

[0033] S41, the encoder consists of the spatial information channels of the multimodal fusion features and the additional conditions as the encoder input, which are mapped together to the latent space to output the parameters of the latent distribution of each key point, the mean vector μ and the variance vector σ, and obtain the latent feature Z=μ+σ·∈, ∈~N(0,I) is the noise sampled from the Gaussian distribution, where the additional conditions are the extra information contained in other channels besides the spatial information channels of the multimodal fusion features;

[0034] S42, construct the decoder, superimpose the latent feature Z and additional conditions into channels to obtain the stitched feature shape, and decode it through a multilayer perceptron (MLP) to obtain the final predicted hand pose.

[0035] Compared with the prior art, the present invention has the following beneficial effects: The visual multimodal non-contact gesture unlocking method of the present invention maps multimodal fusion features to a neural network in the latent space, outputs parameters u and q of the latent distribution, and generates latent features Z through reparameterization techniques. It uses multimodal data to verify the user's identity, solves the limitations of single modality, the uncertainty of the user's hand distribution and the difficulty in capturing its patterns caused by the complexity of the environment, reduces the false recognition rate and improves the robustness of the system, and improves the security and reliability of the electronic lock system.

[0036] This invention presents a visual multimodal contactless gesture unlocking method, proposing a gesture recognition approach based on the fusion of event streams and RGB images. By constructing a spatiotemporal graph structure, it achieves accurate modeling of hand movements. This method innovatively uses keypoints extracted from a pre-trained hand pose estimation model as vertices, fusing multidimensional attributes such as shape, position, and motion state, and constructs temporal associations by dynamically connecting consecutive frames through event streams. It explicitly models the gesture evolution process using a hierarchical graph structure and adaptively adjusts edge weights using event stream timestamps, effectively solving the problem of temporal information loss in fast-moving scenarios common in traditional methods. Through dynamic spatial topology weighting and timestamp weighting mechanisms, it simultaneously optimizes hand structure representation and motion trend capture. Combining a bi-branch graph convolutional network, it deeply fuses the static features of RGB images with the high-precision dynamic information of event streams, significantly improving the accuracy and real-time performance of gesture recognition in complex scenes. It overcomes the limitations of information redundancy and temporal mismatch in traditional multimodal fusion, providing a low-latency, highly robust real-time gesture interaction and high-precision sign language recognition suitable for challenging environments such as changing lighting and rapid movement.

[0037] The visual multimodal non-contact gesture unlocking method of the present invention utilizes multimodal joint prediction and dynamic key generation and transmission to build a more secure and reliable electronic lock system, which has great application prospects. Attached Figure Description

[0038] Figure 1 This is a system flowchart of the visual multimodal non-contact gesture unlocking method of the present invention;

[0039] Figure 2 This is a schematic diagram illustrating the calling principle of the electronic device module in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram illustrating the training principle of the gesture recognition model according to an embodiment of the present invention.

[0041] Figure 4 This is a schematic diagram of the electronic device structure for the gesture unlocking method according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.

[0043] Example 1

[0044] A visual multimodal contactless gesture unlocking method, such as Figures 1-3 As shown, where Figure 1 This is a flowchart of the method system of the present invention; Figure 2 This is a schematic diagram illustrating the calling principle of the electronic device module in an embodiment of the present invention; Figure 3This is a schematic diagram illustrating the training principle of the gesture recognition model according to an embodiment of the present invention. By combining infrared ranging, event streaming, RGB images, and feature fusion technology, high-precision gesture recognition and unlocking operations are achieved, which includes the following feature steps:

[0045] S1. Acquisition of multi-source sensor data: The infrared ranging sensor module detects the approaching hand movement, activates and schedules the multi-source camera module to acquire hand data.

[0046] The preprocessing of S2.RGB image data involves determining the sampling points of the RGB frame position channel and color channel using an offset matrix. The position of each sampling point is dynamically adjusted based on the feature map, and the sampling area is reweighted to amplify the channels that express stronger hand features, thus obtaining preliminary hand features.

[0047] S3. The fusion of event stream and RGB image: Based on the initially obtained hand features, the features of fingers and palm are used as vertices and the connecting lines are used as edges to express the spatial structure of the hand. The connecting lines of the same vertices in consecutive frames are used as edges to construct the temporal correlation. Then, the event stream is dynamically mapped to the graph structure to obtain the multimodal fusion features.

[0048] In step S3, the joints extracted by the pre-trained hand pose estimation model are used as vertices. Multi-dimensional attributes such as shape, position and motion state are fused. Temporal associations are constructed by dynamically connecting consecutive frames through event streams. The gesture evolution process is explicitly modeled using a hierarchical graph structure. The edge weights are adaptively adjusted using event stream timestamps. Through dynamic spatial topology weighting and timestamp weighting mechanisms, the hand structure representation and motion trend capture are optimized at the same time. Combined with a two-branch graph convolutional network, the static features of RGB images and the high-precision dynamic information of event streams are deeply fused.

[0049] S4. Multimodal joint prediction feeds multimodal fusion features into a conditional variational autoencoder to identify hand poses and convert them into computer-understandable text sequences.

[0050] In step S1, during data acquisition, multiple RGB-D cameras with different perspectives and an event camera are used to capture some gesture data as real data. The RGB images are first adaptively interpolated and then a synthetic event stream is generated through an event simulator as synthetic data, and the ground truth is constructed.

[0051] The specific implementation details of step S2 are as follows:

[0052] S21. An offset matrix is ​​computed using a learnable offset module. This module predicts the required sampling offset for each location using a lightweight neural network, allowing the convolutions to focus on key regions of the hand (e.g., fingertips, palm contours, etc.). The output is an offset tensor of shape ijk2 (k2 offset points for each pixel). This offset dynamically adjusts the sampling position of each convolution kernel, making the kernel shape adaptable to the contours of the hand. The offset sampling region is no longer a regular grid, but rather shifted to a local region that better captures key features, reducing unnecessary computation and parameter count. The pixel values ​​of the offset points are calculated using bilinear interpolation, and feature one is obtained using regular convolution.

[0053] S22. Split the feature map into channels, calculate the global average value of each channel, generate the weight of each channel through an activation function, and then reweight the channels of the original feature map using learnable parameters. This amplifies the channels that better express the hand features, while suppressing the channels that suppress background noise. The preprocessed hand feature map is obtained.

[0054] The specific implementation details of step S3 are as follows:

[0055] Step S3 specifically includes the following steps:

[0056] S31 constructs the vertices of the graph, using key hand feature regions such as fingertips, palms, and joints as vertices, and incorporating the output of a pre-trained model as vertex attributes to form a multi-dimensional vertex representation. By introducing the output of the pre-trained model, each vertex is assigned multiple attributes, including shape, position, and motion state. This allows the graph vertices to not only represent the static structure of the hand but also contain rich dynamic information. This multi-dimensional representation method can more accurately capture the complex changes in gestures, solving the problem that gesture recognition often relies on image processing or simple keypoint detection, which cannot fully capture the details of dynamic changes. Our method can effectively handle complex gestures and subtle changes through multi-dimensional vertex attributes, making recognition more accurate and robust in dynamic contexts.

[0057] S32, constructing the temporal layers of the graph, uses consecutive frames in time as graph layers. Each frame corresponds to a hand graph structure, and the temporal layers are dynamically connected through event streams to achieve the transfer of temporal information. By organizing consecutive frames into graph layers and dynamically connecting them through event streams, effective transfer of temporal information is achieved, fully capturing the temporal characteristics of gesture evolution. This solves the problem of traditional methods lacking explicit modeling of temporal information, resulting in low recognition accuracy for fast gestures or complex actions, and failing to accurately model dynamic changes in the temporal dimension, especially prone to distortion when processing fast and complex gestures. This invention, through dynamic temporal layer connections, ensures that each frame carries complete temporal information, improving the system's ability to capture continuous actions and rapid changes.

[0058] S33, constructing spatial structural connections: In the same time frame, connecting hand feature vertices as edges to construct a spatial structure, and dynamically adjusting the weights of the edges according to the spatial topological relationship of the gesture to enhance the expressive power of the spatial structure; by dynamically adjusting the weights of the edges, the relative positions and movements between different parts of the hand can be flexibly expressed according to the spatial topological relationship of the gesture, providing a more accurate representation of the spatial structure. This solves the problem that existing methods rarely effectively express the relative motion and spatial relationship between different parts of the hand. This invention introduces a flexible weight adjustment mechanism in spatial modeling, which can more accurately identify the dynamic changes of the hand in complex gestures, especially showing extremely high accuracy in subtle gesture transitions or complex hand movements.

[0059] S34, Constructing Temporal Correlation: In adjacent frames, vertices at the same position are connected as edges to construct temporal correlations. These correlations are then weighted using timestamp information from the event stream, improving the accuracy of time series modeling. By utilizing the timestamp information from the event stream to weight the temporal correlations between each frame, the temporal details of gesture actions are accurately captured. This method dynamically adjusts temporal relationships based on the timestamp of each frame, avoiding time error problems that traditional methods cannot handle. This invention, through weighted temporal correlation, can accurately capture minute changes in time, improving the accuracy of time series modeling and making the system more sensitive and precise when processing dynamic gestures.

[0060] S35, Multimodal Fusion and Graph Convolution, synchronizes the timestamps of the event stream with RGB frames. By accumulating the positive and negative channels of the event stream, a time window matching the RGB frames is generated. This time window is dynamically mapped to a graph structure stored in an adjacency matrix. Graph convolution is performed on the graph structure to extract multimodal fusion features, with each channel containing temporal, spatial, and motion pattern information. This method fuses multimodal data from the event stream and RGB images, and performs in-depth analysis of the graph structure through graph convolution. This fusion method fully utilizes the high-precision temporal information provided by the event stream and the spatial information provided by the RGB images, effectively improving the comprehensive capabilities of gesture recognition. It solves the problem of effectively fusing multimodal data, especially in complex environments where the coordination between RGB images and event streams often leads to information redundancy or mismatch. This invention, by dynamically mapping the time window and combining it with the graph structure, effectively solves the problem that traditional techniques cannot simultaneously process temporal, spatial, and motion pattern information, greatly improving the accuracy and adaptability of recognition.

[0061] In S31, the adjacency matrix is ​​used to store potential spatiotemporal relationships. First, the key feature regions of the hand (such as fingertips, knuckles, and the center of the palm) are used as vertices of the graph.

[0062] In S32, consecutive frames in time are used as layers of the graph to ensure that each frame contains a hand graph structure.

[0063] In S33, spatial structural connections are constructed: in the same time frame, the vertices connecting hand features are stored as edges (e.g., connections from the palm to each finger).

[0064] In S34, temporal association is constructed: vertices at the same position in adjacent frames are stored as edges (such as the fingertip in the current frame and the fingertip in the next frame).

[0065] In S35, the timestamp of the event stream is synchronized with the RGB frame. A time window matching the RGB frame is generated by accumulating the positive and negative dual channels of the event. It is dynamically mapped to the graph structure stored in the adjacency matrix. The graph convolution operation is performed to obtain multi-modal fusion multi-channel features. Each channel contains information such as temporal and spatial motion modes.

[0066] The pre-trained model is a deep learning framework that integrates convolutional neural networks (CNNs) and pose regression networks. It is trained end-to-end on large-scale public hand keypoint datasets (such as FreiHAND or Hand-143K) to achieve high-precision regression and recognition of the three-dimensional spatial coordinates and category attributes of hand keypoints (such as fingertips, joints, and palms). Specifically, the joints extracted by the pre-trained model from the input RGB images and event stream images are used as vertices of the graph structure. Each vertex is assigned multi-dimensional attributes, including its position vector in three-dimensional space, the shape feature descriptor of the current frame (such as the degree of finger opening and the convex hull area of ​​the palm region), and the motion state encoding based on the model output (such as category labels such as stationary, sliding, and rotating). It can be further extended to add the historical trajectory point features of the vertex in the most recent time sequence to form a multi-dimensional spatiotemporal and semantic fusion vertex representation, realizing multi-angle perception and expression of hand state.

[0067] The specific calculation methods for each dimension of the vertex multidimensional attributes are as follows: Shape attributes are obtained by binarizing and segmenting the hand region in each frame, and combining the Euclidean distance, angle between key points, and regional geometric features (such as convex hull area, finger angle, palm width, etc.). Principal component analysis (PCA) or an autoencoder is then used to reduce the dimensionality and generate a compact shape feature vector. Motion state is automatically determined by analyzing the time difference and velocity vector of the vertex spatial position in consecutive frames, and identifying motion modes such as stationary, translation, rotation, and opening / closing. Features can be described using Δx, Δy, Δz, and direction cosine, etc. The pre-trained gesture classification attribute directly calls the gesture recognition model to classify the global gesture type of the current frame or time segment, and appends the category probability distribution vector to the vertex attributes. The above multidimensional attributes are finally concatenated into a high-dimensional feature vector, which comprehensively represents the shape, motion, and category features of the vertex.

[0068] The specific synchronization method for the event flow to connect consecutive frames is as follows: First, through the hardware time synchronization mechanism, the time bases of the event camera and the RGB-D camera acquisition module are unified to ensure that the timestamps of the event stream are strictly aligned with the acquisition time points of each RGB frame. Subsequently, within each RGB frame acquisition window, all events in the event stream that overlap with the window time are intercepted and divided into two channels, the positive channel (brightness increase) and the negative channel (brightness decrease), according to the event timestamps. Cumulative statistics are performed separately to generate the positive and negative event count maps corresponding to the RGB frame time window. The algorithm process is as follows: For each frame [t, t + Δt], count all events in the event stream E = {ei=(xi, yi, ti, pi)} that satisfy t ≤ ti < t + Δt, and accumulate the quantities of the two polarities pi = +1 and pi = -1 respectively to form positive and negative event window mapping graphs in the form of W+(x, y) and W-(x, y). This cumulative result further participates in the graph structure construction and multi-modal feature fusion to achieve precise alignment and information enhancement of the event stream and RGB frames in space and time.

[0069] The specific method for the event stream timestamp to adaptively adjust the edge weights is as follows: When constructing the graph structure, for each edge between a pair of vertices vi and vj, its spatial weight ws is calculated by the Gaussian kernel function according to the spatial distance dij between the vertices, ws = exp(-dij2 / 2σs2), where σs is the spatial scale parameter; the time weight wt is adaptively adjusted according to the event stream timestamp interval Δtij, and a weight assignment method similar to temporal decay is adopted, wt = exp(-Δtij / τ), where τ is the time scale hyperparameter; the total weight Wij of the edge is Wij = α·ws + β·wt, where α and β are adjustable weighting coefficients that can be adjusted according to the importance of spatial or temporal associations; the spatially variable weight automatically increases / decreases the spatial weight by real-time detecting changes in the vertex spatial topology (such as fingers quickly separating or closing), and the temporally variable weight dynamically fine-tunes the time weight in combination with the event stream burst intensity or temporal local change trend to enhance the model's sensitivity to drastic spatial structure changes and temporal dynamic evolution.

[0070] The specific structure for performing graph convolution operations on the graph structure is a dual-branch dynamic graph convolutional network, which processes the adjacency matrix and node features from RGB static features and event stream dynamic features respectively. Each branch contains multiple layers of spatial graph convolution (such as GCN, GraphSAGE, or GAT structures), which realize cross-channel propagation of spatial and temporal information in each layer, and introduce feature normalization and nonlinear activation after each layer. The multimodal feature fusion mechanism is an interactive attention mechanism and feature concatenation in parallel. First, attention-weighted fusion of the two modal features is performed, and then concatenation is performed in the fusion layer and passed to the downstream MLP decoder. The process of mapping the time window to the adjacency matrix is ​​as follows: the positive and negative event graphs of the accumulated event stream in each time window are used as weights to update the corresponding element Aij(t) in the adjacency matrix A(t) in real time. That is, if the event intensity in the window is greater than the threshold, the weight of the relevant edge is strengthened, and vice versa. This process is automatically triggered by the gating mechanism or threshold discrimination to realize the dynamic evolution of the adjacency matrix in time and the injection of multimodal information, ensuring that the graph structure can continuously and adaptively reflect the changes in hand state and movement.

[0071] The specific implementation details of step S4 are as follows:

[0072] S41. Construct an encoder. The encoder takes the spatial information channels of the multimodal fusion features as input, and uses the additional information contained in other channels as "additional conditions" to map them together into the latent space. It outputs the parameters of the latent distribution of each key point, the mean vector μ, and the variance vector σ. The latent feature Z = μ + σ ∈, where ∈ ~ N(0, I) is noise sampled from the Gaussian distribution.

[0073] S42. Construct a decoder that overlays the features Z sampled from the latent space and the "additional conditions" into channels, and then passes them through an MLP to obtain the final predicted hand pose.

[0074] The implementation details of the deep learning algorithm in this embodiment include the following steps:

[0075] S1. Forward propagation steps:

[0076] Input preprocessing: The input RGB image is pre-processed through a neural network to dynamically adjust the local transformation parameters of the hand region, ensuring that the model can focus on the key areas of the hand.

[0077] Spatiotemporal feature alignment: The event stream and image features are input into a spatiotemporally related neural network module. The temporal and spatial relationships of hand joints are modeled through graph structures to achieve temporal and spatial alignment between the event stream and image features.

[0078] Joint inference: The fused features are input into the encoder and jointly inferred by the decoder to predict and estimate the three-dimensional position information of the hand node in the world coordinate system.

[0079] S2. Calculate the loss using three loss functions:

[0080] Position loss: The difference between the predicted hand joint position and the true position is measured using mean squared error (MSE).

[0081] Reconstruction loss: The reconstruction loss is used to ensure that the hand pose predicted by the decoder is as close as possible to the true pose.

[0082] KL divergence loss: Adding KL divergence loss normalizes the distribution of the latent space, improving the stability and generalization ability of the generative model.

[0083] S3. Backpropagation optimization:

[0084] Optimizer selection: Adaptive optimization algorithms, such as the Adam optimizer, are used to accelerate convergence and improve training efficiency.

[0085] Learning rate scheduling: The learning rate is dynamically adjusted according to the training progress, and a learning rate decay strategy is adopted to prevent the model from getting stuck in local optima in the later stages of training.

[0086] Regularization techniques: L2 regularization and Dropout layers are introduced to prevent model overfitting and improve its generalization ability.

[0087] Batch Normalization: Using batch normalization between layers accelerates the training process and stabilizes model performance.

[0088] The specific steps for training and deploying the deep learning model are as follows:

[0089] Data acquisition utilizes multiple-view RGB-D cameras and an event camera to capture some gesture data as real data. RGB images are first adaptively interpolated and then synthesized into a synthetic event stream through an event simulator as synthetic data, and ground truth is constructed, aiming to provide data support for deep learning.

[0090] Model training fully utilizes deep learning technology to finely train the collected gesture data in order to build an efficient and accurate hand gesture recognition model.

[0091] Model optimization involves continuous iterative optimization of the model and the addition of data augmentation to improve the accuracy of gesture intent recognition.

[0092] Model deployment involves deploying the trained model to the embedded recognition module, giving it powerful hand gesture recognition capabilities.

[0093] The gesture recognition model described in this embodiment includes a hand recognition module and a recording module. The hand recognition module is responsible for recognizing and tracking hand gestures and converting them into a key sequence that is easy for a computer to understand. The recording module is responsible for recording and storing the "password".

[0094] Gesture sequences may include, but are not limited to, common continuous hand movements such as stretching, grasping, making number gestures, pinching gestures, and flipping gestures. These hand gesture sequences can directly control the opening and closing of the target electronic lock module.

[0095] The method in this embodiment is applied to the following target electronic locks:

[0096] Mechanical locks are switch locks with mechanical structures based on embedded chips or smart circuits;

[0097] Software application lock. A virtual application lock based on the operating system.

[0098] The electronic device module of this embodiment is characterized by comprising an infrared ranging sensor, an event camera, an RGB camera, a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. The infrared ranging sensor, the event camera, the RGB camera, and the processor are connected via a bus, and the processor is coupled to the memory. When the processor executes the computer program, it implements the aforementioned visual multimodal contactless gesture unlocking method.

[0099] The above specific embodiments are used to explain and illustrate the present invention, and are only preferred embodiments of the present invention, not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A visual multimodal contactless gesture unlocking method, characterized in that: Includes the following steps: S1. Acquisition of multi-source sensor data: The infrared ranging sensor module is used to detect the approaching hand movement, triggering and scheduling the multi-source camera module to acquire hand data, obtaining hand movement data with high dynamic range and rich texture information. The preprocessing of S2.RGB image data involves determining the sampling points of the RGB frame position channel and color channel through the offset matrix, dynamically adjusting the position of each sampling point, and reweighting the sampling area to amplify the channels that express hand features more strongly, thus initially extracting hand features. S3. The fusion of event stream and RGB image: Based on the initial extraction of hand features, the features of fingers and palm are used as vertices and the connecting lines are used as edges to express the spatial structure of the hand; the temporal correlation is constructed by connecting the same vertices in consecutive frames as edges, and then the event stream is dynamically mapped to the graph structure to obtain the multimodal fusion features. S4. Multimodal joint prediction maps multimodal fusion features to a neural network model in the latent space, outputs the parameters μ and σ of the latent distribution, and generates latent features Z through reparameterization techniques; The latent feature Z is superimposed with the additional condition channel and then decoded by a multilayer perceptron (MLP) to obtain the final hand pose prediction. In step S3, the joints extracted by the pre-trained hand pose estimation model are used as vertices. Multi-dimensional attributes such as shape, position and motion state are fused. Temporal associations are constructed by dynamically connecting consecutive frames through event streams. The gesture evolution process is explicitly modeled using a hierarchical graph structure. The edge weights are adaptively adjusted using event stream timestamps. Through dynamic spatial topology weighting and timestamp weighting mechanisms, the hand structure representation and motion trend capture are optimized at the same time. Combined with a two-branch graph convolutional network, the static features of RGB images and the high-precision dynamic information of event streams are deeply fused.

2. The visual multimodal non-contact gesture unlocking method as described in claim 1, characterized in that: In step S1, during data acquisition, multiple RGB-D cameras with different perspectives and an event camera are used to capture some gesture data as real data. The RGB images are first adaptively interpolated and then a synthetic event stream is generated through an event simulator as synthetic data, and the ground truth is constructed.

3. The visual multimodal non-contact gesture unlocking method as described in claim 1, characterized in that, Step S2 specifically includes the following steps: S21: A lightweight neural network predicts the sampling offset for each pixel position, generating an offset tensor of shape ijk2. The offset dynamically adjusts the sampling position of the convolution kernel to focus on the key hand region. Bilinear interpolation is used to calculate the pixel value at the offset point. Regular convolution is then performed on the offset sampling region to obtain feature one. S22, split feature 1 by channel, calculate the global average value of each channel, generate the weight of each channel through the activation function, and reweight the channels of the original feature map using learnable parameters to enhance the channels that express hand features, suppress background noise, and obtain the preprocessed hand feature map.

4. The visual multimodal non-contact gesture unlocking method according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31, Construct the vertices of the graph, taking the key feature regions of the hand, including fingertips, palms, and joints, as the vertices of the graph, and introducing the output of the pre-trained model as vertex attributes to form a multi-dimensional vertex representation; S32, construct the time layer of the graph, take consecutive frames in time as the graph layer, each frame corresponds to a hand graph structure, and the time layers are dynamically connected through event flow to realize the transmission of time sequence information; S33: Construct spatial structure connections. In the same time frame, connect the vertices of hand features as edges to construct a spatial structure, and dynamically adjust the weights of the edges according to the spatial topological relationship of the gesture to enhance the expressive power of the spatial structure. S34, Constructing Temporal Correlation: In adjacent frames, vertices at the same position are connected as edges to construct temporal correlation, and the temporal correlation is weighted by the timestamp information of the event stream to improve the modeling accuracy of time series. S35, multimodal fusion and graph convolution, synchronizes the timestamps of the event stream with the RGB frames, generates a time window that matches the RGB frames by accumulating the positive and negative dual channels of the event stream, dynamically maps the time window to the graph structure stored in the adjacency matrix, performs graph convolution operation on the graph structure, and extracts multi-channel features of multimodal fusion. Each channel contains temporal, spatial and motion mode information.

5. The visual multimodal non-contact gesture unlocking method according to claim 4, characterized in that: The pre-trained model is a deep learning framework that integrates a convolutional neural network (CNN) and a pose regression network. It is trained end-to-end on a large-scale public hand keypoint dataset to achieve high-precision regression and recognition of the three-dimensional spatial coordinates and category attributes of hand keypoints. Specifically, the keypoints extracted by the pre-trained model from the input RGB image and event stream image are used as vertices of the graph structure. Each vertex is given multi-dimensional attributes, including its position vector in three-dimensional space, the shape feature descriptor of the current frame, and the motion state encoding based on the model output. The historical trajectory point features of the vertex in the most recent time sequence are further extended to form a multi-dimensional spatiotemporal and semantic fusion vertex representation, realizing multi-angle perception and expression of hand state. The pose regression network is HRNet or MediaPipe Hands hand pose estimation model.

6. The visual multimodal non-contact gesture unlocking method according to claim 5, characterized in that: The specific calculation methods for each dimension in the multidimensional attributes of the vertex are as follows: the shape attribute is obtained by binarizing the hand region of each frame, combining the Euclidean distance between key points, the included angle and the geometric features of the region, and further using principal component analysis or autoencoder to reduce the dimension and generate a compact shape feature vector; the motion state is automatically determined by analyzing the time difference and velocity vector of the vertex spatial position in consecutive frames, and the static, translational, rotational and opening / closing motion modes. The features are described by Δx, Δy, Δz and direction cosine; the pre-trained gesture classification attributes directly call the gesture recognition model to classify the global gesture type of the current frame or time segment, and attach the category probability distribution vector to the vertex attributes; the above multi-dimensional attributes are finally concatenated into a high-dimensional feature vector, which comprehensively represents the shape, motion and category features of the vertex.

7. The visual multimodal non-contact gesture unlocking method according to claim 1, characterized in that: The specific synchronization method for the dynamic connection of continuous frames in the event stream is as follows: First, the time base of the event camera and the RGB-D camera acquisition module is unified through a hardware time synchronization mechanism to ensure that the timestamp of the event stream is strictly aligned with the acquisition time point of each RGB frame. Subsequently, within each RGB frame acquisition window, all events overlapping with the time of that window are captured from the event stream. These events are then divided into positive and negative channels based on their timestamps, and cumulative statistics are performed on each channel to generate a positive and negative event count map for the corresponding RGB frame time window. The algorithm is as follows: For each frame [t, t+Δt], the event stream E={e i =(x i y i , t i p i All in )} satisfy t≤t i For events less than t+Δt, accumulate p respectively. i =+1 and p i =-1, the number of two polarities, forming a shape like W. + (x, y), W - The positive and negative event window mapping graph of (x, y) is further involved in graph structure construction and multimodal feature fusion to achieve accurate spatial and temporal alignment and information enhancement of event stream and RGB frame.

8. The non-contact gesture unlocking method with visual multimodality according to claim 1, characterized in that: The specific method for adaptively adjusting edge weights using event stream timestamps is as follows: When constructing the graph structure, for each pair of vertices v i With v i The spatial weight w of the edge between them s The spatial distance d between vertices ij Calculated using the Gaussian kernel function, w s =exp(-d ij 2 / 2σ s 2 ), where σ s Spatial scale parameter; time weight w t Based on the event stream timest interval Δt ij Adaptive adjustment, using a time-decaying weight allocation method, w t =exp(-Δt ij / τ), where τ is the time scale hyperparameter; the total weight W of the edge. ij =α·w s +β·w t α and β are adjustable weighting coefficients that can be adjusted according to the importance of spatial or temporal correlations. Spatial variable weights automatically increase / decrease spatial weights by detecting changes in vertex spatial topology in real time, while temporal variable weights dynamically fine-tune temporal weights by combining the burst intensity of event flow or the trend of local temporal changes, thereby improving the model's sensitivity to dramatic changes in spatial structure and dynamic evolution over time.

9. The visual multimodal non-contact gesture unlocking method according to claim 4, characterized in that: The specific structure for performing graph convolution operations on the graph structure is to use a dual-branch dynamic graph convolutional network to process the adjacency matrix and node features from RGB static features and event stream dynamic features, respectively. Each branch contains multiple layers of spatial graph convolution, which realizes cross-channel propagation of spatial and temporal information in each layer, and introduces feature normalization and non-linear activation after each layer. The multimodal feature fusion mechanism combines interactive attention and feature concatenation in parallel. First, attention-weighted fusion of features from two modalities is performed. Then, the features are concatenated at the fusion layer and passed to the downstream MLP decoder. The process of mapping the time window to the adjacency matrix is ​​as follows: using the accumulated positive and negative event graphs of the event stream within each time window as weights, the corresponding element A in the adjacency matrix A(t) is updated in real time. ij (t), that is, if the intensity of an event within the window is greater than the threshold, the weight of the relevant edge is increased, and vice versa; this process is automatically triggered by a gating mechanism or threshold discrimination, so as to realize the dynamic evolution of the adjacency matrix in time and the injection of multimodal information, ensuring that the graph structure can continuously and adaptively reflect the changes in hand state and action.

10. The visual multimodal non-contact gesture unlocking method according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41, the encoder consists of spatial information channels and additional conditions of multimodal fusion features as input, which are mapped together to the latent space. The output parameters of the latent distribution at each key point are the mean vector μ and the variance vector σ, thus obtaining the latent features. It is noise sampled from a Gaussian distribution, where the additional condition is that other channels besides the spatial information channel of the multimodal fusion feature contain additional information; S42, construct the decoder, superimpose the latent feature Z and additional conditions into channels to obtain the stitched feature shape, and decode it through a multilayer perceptron (MLP) to obtain the final predicted hand pose.

Images