Visual multi-mode non-contact gesture unlocking method

Abstract

The invention discloses a visual multi-mode non-contact gesture unlocking method. The method comprises the following steps: S1, collecting multi-source sensing data; s2, preprocessing the RGB image data; s3, the event stream and the RGB image are fused, the features of the fingers and the palm are used as vertexes, connecting lines are used as edges, and a hand space structure is expressed; constructing time correlation by taking a connecting line of the same vertexes in front and back continuous frames as an edge, and dynamically mapping an event stream into a graph structure to obtain a multi-modal fusion feature; s4, performing multi-modal joint prediction, mapping the multi-modal fusion feature to a neural network model of a potential space, outputting parameters mu and sigma of potential distribution, and generating a potential feature Z through a re-parameterization technique; and after the potential feature Z and the additional condition channel are overlapped, decoding is carried out through a multi-layer perceptron (MLP), and final hand posture prediction is obtained. The invention provides low-delay and high-robustness real-time gesture interaction and high-precision sign language recognition suitable for challenging environments such as illumination variation and fast movement.

Description

Technical Field

[0001] The present invention belongs to the technical field of gesture tracking and recognition, and in particular relates to a visual multimodal contactless gesture unlocking method. Background Art

[0002] With the rapid development of science and technology, people's demand for safety and convenience in life continues to increase, and the importance of public health and safety has become increasingly prominent. Traditional mechanical locks and contact electronic locks are unable to meet the high standards of modern safety and hygiene, which has promoted the rapid development of contactless unlocking technology. Existing contactless unlocking technologies mainly include radio frequency identification (RFID), near-field communication (NFC), biometrics (such as iris recognition and facial recognition), and gesture recognition based on machine vision. Although these technologies have been widely used in smart door locks, mobile phone unlocking, and electronic payments, they still have many limitations:

[0003] Radio Frequency Identification (RFID) and Near Field Communication (NFC): They are susceptible to electromagnetic interference, resulting in unstable unlocking effects and low reliability in complex environments.

[0004] Biometric technology: Although it has high security, it still needs to be improved in terms of recognition speed and system scalability, and its recognition performance is limited under certain lighting conditions.

[0005] Machine vision-based gesture recognition: Traditional cameras perform poorly due to high power consumption, high latency, and low dynamic range, limiting their application in real-time and dynamic environments. Furthermore, existing vision systems struggle to maintain a balanced recognition accuracy and response speed in complex backgrounds or when gestures change rapidly.

[0006] Single modalities such as RGB images have insufficient recognition accuracy under complex lighting and background interference, limited dynamic range leads to loss of details, high computational complexity makes it difficult to meet real-time requirements, lack effective capture of time correlation, background noise interference affects recognition accuracy, key generation and transmission process has low security, and there is a lack of effective user feedback mechanism when verification fails. These problems limit the accuracy, security, real-time and user experience of gesture recognition technology.

[0007] Therefore, how to solve the problem of simultaneous acquisition of high dynamic range and rich texture information, ensure effective weighting of key features during the sampling process to highlight hand features and suppress background noise, build a system architecture that can accurately capture and track the user's continuous action sequences, and convert these data into text form and encrypt and save them for verification are technical problems that need to be urgently solved by technical personnel in this field. Summary of the Invention

[0008] The purpose of the present invention is to provide a visual multimodal contactless gesture unlocking method to address the problems in the prior art.

[0009] To this end, the above-mentioned purpose of the present invention is achieved through the following technical solutions:

[0010] A visual multimodal contactless gesture unlocking method, characterized by comprising the following steps:

[0011] S1. Multi-source sensor data acquisition: Using an infrared ranging sensor module to sense approaching hand movements, triggering and scheduling multi-source camera modules to collect hand data, obtaining hand motion data with high dynamic range and rich texture information;

[0012] S2. Preprocessing of RGB image data: Determine the sampling points of the RGB frame position and color channels using an offset matrix, dynamically adjust the position of each sampling point, and reweight the sampling area to amplify channels that more strongly express hand features, thereby initially extracting hand features.

[0013] S3. Fusion of event streams and RGB images. Based on preliminary hand feature extraction, the spatial structure of the hand is expressed using finger and palm features as vertices and connecting lines as edges. Temporal associations are constructed by connecting the same vertices in consecutive frames as edges. The event stream is then dynamically mapped onto a graph structure to obtain multimodal fusion features.

[0014] S4. Multimodal joint prediction: This maps the multimodal fusion features to a neural network model in the latent space, outputs the parameters μ and σ of the latent distribution, and generates the latent feature Z through reparameterization techniques. The latent feature Z is superimposed with the additional conditional channel and decoded by a multi-layer perceptron (MLP) to obtain the final hand posture prediction.

[0015] In step S3, the joint points extracted by the pre-trained hand posture estimation model are used as vertices, and the multi-dimensional attributes of shape, position and motion state are integrated. The continuous frames are dynamically connected through the event stream to build a temporal association. The gesture evolution process is explicitly modeled using a hierarchical graph structure. The edge weights are adaptively adjusted using the event stream timestamp. Through the dynamic spatial topology weighting and timestamp weighting mechanism, the hand structure representation and motion trend capture are optimized at the same time. Combined with the dual-branch graph convolutional network, the static features of the RGB image and the high-precision dynamic information of the event stream are deeply integrated. While adopting the above technical solutions, the present invention can also adopt or combine the following technical solutions:

[0016] As a preferred technical solution of the present invention: in step S1, during data collection, partial gesture data is captured as real data with the help of multi-view RGB-D cameras and an event camera, and the RGB image is first adaptively interpolated and then a synthetic event stream is generated as synthetic data through an event simulator to construct the true value.

[0017] As a preferred technical solution of the present invention: Step S2 specifically includes the following steps:

[0018] S21, through a lightweight neural network to predict the sampling offset of each pixel position, generate an offset tensor of shape ijk2, the offset dynamically adjusts the sampling position of the convolution kernel to focus on the key area of ​​the hand, uses bilinear interpolation to calculate the pixel value of the offset point, and performs conventional convolution on the offset sampling area to obtain feature one.

[0019] S22, split the feature one by channel, calculate the global average of each channel, generate the weight of each channel through the activation function, use the learnable parameters to reweight the channels of the original feature map, enhance the channels expressing the hand features, suppress background noise, and obtain the preprocessed hand feature map.

[0020] As a preferred technical solution of the present invention: Step S3 specifically includes the following steps:

[0021] Step S3 specifically includes the following steps:

[0022] S31, construct the vertices of the graph, taking the key feature areas of the hand such as fingertips, palms, joints, etc. as the vertices of the graph, and introducing the output results of the pre-trained model as vertex attributes to form a multi-dimensional vertex representation;

[0023] S32, constructing the time layer of the graph, using temporally consecutive frames as the graph layer. Each frame corresponds to a hand graph structure. The time layers are dynamically connected through event streams to achieve the transmission of temporal information.

[0024] S33, building spatial structure connections, connects hand feature vertices as edges to build spatial structures in the same time frame, and dynamically adjusts the weights of edges according to the spatial topological relationship of gestures to enhance the expressiveness of spatial structures;

[0025] S34, build temporal associations: In adjacent frames, connect vertices at the same position as edges to build temporal associations. The temporal associations are weighted using the timestamp information of the event stream to improve the modeling accuracy of the time series.

[0026] S35, multimodal fusion and graph convolution, synchronizes the timestamp of the event stream with the RGB frame, generates a time window that matches the RGB frame through the positive and negative dual-channel accumulation of the event stream, dynamically maps the time window to the graph structure stored in the adjacency matrix, performs graph convolution operations on the graph structure, and extracts multi-channel features of multimodal fusion. Each channel contains time, space, and motion pattern information.

[0027] As a preferred technical solution of the present invention: The pre-trained model is a deep learning framework that integrates a convolutional neural network (CNN) and a pose regression network. Through end-to-end training on a large-scale publicly available hand keypoint dataset, high-precision regression and recognition of the three-dimensional spatial coordinates and class attributes of hand keypoints are achieved. Specifically, the keypoints extracted by the pre-trained model from the input RGB image and event stream image are used as the vertices of the graph structure. Each vertex is simultaneously assigned multi-dimensional attributes, including: its position vector in three-dimensional space, the shape feature descriptor of the current frame, and the motion state encoding based on the model output. Furthermore, the historical trajectory point features of the vertex within the recent time series can be extended and added to form a multi-dimensional spatio-temporal and semantic fusion vertex representation, realizing multi-angle perception and expression of the hand state. Among them, the pose regression network is the HRNet or the MediaPipe Hands hand pose estimation model.

[0028] As a preferred technical solution of the present invention: The specific calculation methods for each dimension in the multi-dimensional attributes of the vertex are as follows: The shape attribute is calculated by performing binary segmentation on the hand region of each frame, combining the Euclidean distance, angle, and regional geometric features between keypoints, and further using principal component analysis or autoencoder for dimensionality reduction to generate a compact shape feature vector; The motion state is analyzed through the time difference and velocity vector of the vertex spatial position in consecutive frames to automatically distinguish action modes such as static, translation, rotation, opening and closing, etc. The features can be described by, for example, Δx, Δy, Δz, and direction cosine; The pre-trained gesture classification attribute directly calls the gesture recognition model to classify the global gesture type of the current frame or time series segment, and attaches the class probability distribution vector to the vertex attribute; The above multi-dimensional attributes are finally concatenated into a high-dimensional feature vector to comprehensively represent the shape, motion, and class features of the vertex.

[0029] As a preferred technical solution of the present invention: The specific synchronization method for the event flow to connect consecutive frames is as follows: First, through the hardware time synchronization mechanism, the time bases of the event camera and the RGB-D camera acquisition module are unified to ensure that the time stamps of the event stream are strictly aligned with the acquisition time points of each RGB frame; Subsequently, within each RGB frame acquisition window, all events in the event stream that overlap with the window time are intercepted and divided into two channels, the positive and negative channels, according to the event time stamps, and cumulative statistics are performed respectively to generate the positive and negative event count maps corresponding to the RGB frame time window. The algorithm process is as follows: For each frame [t, t + Δt], count all events in the event stream E = {e i =(x i , y i , t i , p i )} that satisfy t ≤ t i < t + Δt, and accumulate p i = +1 and p i=-1The number of two polarities, forming a W shape + (x, y), W - (x, y) positive and negative event window mapping diagram; the accumulated results are further involved in graph structure construction and multimodal feature fusion to achieve precise alignment and information enhancement between event streams and RGB frames in space and time.

[0030] As a preferred technical solution of the present invention: the specific method of adaptively adjusting the edge weight of the event stream timestamp is: when constructing the graph structure, each pair of vertices v i With v j The edge between them has a spatial weight w s The spatial distance d between vertices ij Calculated by Gaussian kernel function, w s =exp(-d ij 2 / 2σ s 2 ), where σ s is the spatial scale parameter; the time weight w t According to the event stream timestamp interval Δt ij Adaptive adjustment, using a weight distribution method similar to time series decay, w t =exp(-Δt ij / τ), τ is the time scale hyperparameter; the total weight of the edge W ij =α·w s + β·w t , where α and β are adjustable weighting coefficients that can be adjusted according to the importance of spatial or temporal associations; spatially variable weights automatically increase / decrease spatial weights by detecting vertex spatial topology changes in real time, and temporally variable weights dynamically fine-tune temporal weights based on the burst intensity of event flows or local temporal change trends, thereby improving the model's sensitivity to sudden changes in spatial structure and temporal dynamic evolution.

[0031] As the preferred technical solution of the present invention: the specific structure of performing graph convolution operation on the graph structure is to adopt a dual-branch dynamic graph convolution network to process the adjacency matrix and node features from RGB static features and event flow dynamic features respectively; each branch contains multiple layers of spatial graph convolution, and realizes cross-channel propagation of spatial and temporal information in each layer, and introduces feature normalization and nonlinear activation after each layer; the multimodal feature fusion mechanism is an interactive attention mechanism and feature splicing in parallel, firstly, the two modal features are subjected to attention weighted fusion, and then spliced ​​in the fusion layer and passed to the downstream MLP decoder; the process of mapping the time window to the adjacency matrix is: using the positive and negative event graphs of the event flow accumulated in each time window as weights, and updating the corresponding elements A in the adjacency matrix A(t) in real time ij(t), that is, if the event intensity within the window is greater than the threshold, the weight of the related edge is enhanced, otherwise it is weakened; this process is automatically triggered by the gating mechanism or threshold discrimination, realizing the dynamic evolution of the adjacency matrix in time series and the injection of multimodal information, ensuring that the graph structure can continuously and adaptively reflect changes in hand status and movement.

[0032] As a preferred technical solution of the present invention: Step S4 specifically includes the following steps:

[0033] S41, the composition of the encoder, the spatial information channel and additional conditions of the multimodal fusion feature are used as encoder inputs, mapped together to the latent space, and the parameters of the potential distribution of each joint point, the mean vector μ and the variance vector σ, are output to obtain the latent feature Z=μ+σ·∈, ∈~N(0,I) is the noise sampled from the Gaussian distribution, where the additional conditions are the additional information contained in other channels besides the spatial information channel of the multimodal fusion feature;

[0034] S42, construct a decoder, superimpose the latent feature Z and the additional conditions on each channel to obtain the spliced ​​feature shape, and decode it through the multi-layer perceptron MLP to obtain the final predicted hand posture.

[0035] Compared with the prior art, the present invention has the following beneficial effects: the visual multimodal contactless gesture unlocking method of the present invention maps the multimodal features to the neural network of the latent space, outputs the parameters u and q of the latent distribution, and generates the latent feature Z through the re-parameterization technique, and uses multiple modal data to verify the identity of the user, thereby solving the limitations of a single modality, the uncertainty of the distribution of the user's hands and the difficulty in capturing their regularities caused by the complexity of the environment, reducing the misrecognition rate and improving the robustness of the system, thereby improving the security and reliability of the electronic lock system.

[0036] The present invention proposes a visual multimodal contactless gesture unlocking method based on the multimodal fusion of event streams and RGB images. This method accurately models hand movements by constructing a spatiotemporal graph structure. This method innovatively uses joint points extracted from a pretrained hand posture estimation model as vertices, integrating multidimensional attributes such as shape, position, and motion state. It then dynamically connects consecutive frames through the event stream to establish temporal associations. This method explicitly models the gesture evolution process using a hierarchical graph structure and adaptively adjusts edge weights using event stream timestamps, effectively addressing the temporal information loss problem of traditional methods under rapid motion. Dynamic spatial topology weighting and timestamp weighting mechanisms simultaneously optimize hand structure representation and motion trend capture. Combined with a dual-branch graph convolutional network, this method deeply fuses the static features of RGB images with the high-precision dynamic information of the event stream, significantly improving the accuracy and real-time performance of gesture recognition in complex scenarios. This method overcomes the limitations of traditional multimodal fusion, such as information redundancy and temporal mismatch, and provides a low-latency, highly robust method for real-time gesture interaction and high-precision sign language recognition suitable for challenging environments such as changing lighting and rapid motion.

[0037] The visual multimodal contactless gesture unlocking method of the present invention utilizes multimodal joint prediction and dynamic key generation and transmission to build a more secure and reliable electronic lock system, which has great application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a system flow chart of the visual multimodal contactless gesture unlocking method of the present invention;

[0039] Figure 2 A schematic diagram of a calling principle of an electronic device module according to an embodiment of the present invention;

[0040] Figure 3 This is a diagram showing the principles of training a gesture recognition model according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the electronic device structure of the gesture unlocking method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.

[0043] Example 1

[0044] A visual multimodal contactless gesture unlocking method, such as Figure 1-Figure 3 As shown, Figure 1 is a flow chart of the method system of the present invention; Figure 2 is a schematic diagram of a calling principle of an electronic device module according to an embodiment of the present invention; Figure 3This is a diagram showing the principles of training a gesture recognition model in an embodiment of the present invention. By combining infrared ranging, event streams, RGB images, and feature fusion technology, high-precision gesture recognition and unlocking operations are achieved. This includes the following characteristic steps:

[0045] S1. Collection of multi-source sensor data: Using the infrared ranging sensor module to sense the approaching hand movement, activate and schedule the multi-source camera module to collect hand data.

[0046] S2. Preprocessing of RGB image data: The sampling points of the RGB frame position channel and color channel are determined using an offset matrix. The position of each sampling point is dynamically adjusted based on the feature map, and the sampling area is reweighted to amplify channels that express stronger hand features, thereby obtaining preliminary hand features.

[0047] S3. The event stream is fused with the RGB image. Based on the initially obtained hand features, the finger and palm features are used as vertices and the connecting lines are used as edges to express the hand's spatial structure. The same vertices in consecutive frames are connected as edges to construct temporal associations. The event stream is then dynamically mapped to the graph structure to obtain multimodal fusion features.

[0048] In step S3, the joint points extracted by the pre-trained hand posture estimation model are used as vertices, and the multi-dimensional attributes of shape, position and motion state are fused. The continuous frames are dynamically connected through the event stream to construct a temporal association. The gesture evolution process is explicitly modeled using a hierarchical graph structure, and the edge weights are adaptively adjusted using the event stream timestamps. Through the dynamic spatial topology weighting and timestamp weighting mechanisms, the hand structure representation and motion trend capture are simultaneously optimized. Combined with the dual-branch graph convolutional network, the static features of the RGB image and the high-precision dynamic information of the event stream are deeply integrated.

[0049] S4. Multimodal joint prediction: The multimodal fusion features are fed into the conditional variational autoencoder to recognize hand postures and convert them into computer-understandable text sequences.

[0050] In step S1, during data collection, multiple-view RGB-D cameras and an event camera are used to capture partial gesture data as real data. RGB images are first adaptively interpolated and then a synthetic event stream is generated as synthetic data through an event simulator to construct the true value.

[0051] The specific implementation details of step S2 are:

[0052] S21. The offset matrix is ​​calculated by a learnable offset module. This module predicts the sampling offset required for each position through a lightweight neural network, so that the convolution can focus on the key areas of the hand (such as fingertips, palm contours, etc.), and the output is an offset tensor of shape ijk2 (there are k2 offset points for each pixel). This offset dynamically adjusts the position of each convolution kernel sample so that the shape of the convolution kernel can adapt to the contour of the hand. The offset sampling area is no longer a regular grid, but is offset to a local area that can better capture key features, reducing unnecessary calculations and parameters. The pixel value of the offset point is calculated by bilinear interpolation, and feature one is obtained using conventional convolution.

[0053] S22. Split the feature map into channels, calculate the global average of each channel, generate a weight for each channel using an activation function, and then reweight the channels of the original feature map using learnable parameters. This amplifies channels that are more representative of hand features and suppresses channels that are more representative of background noise. This yields the preprocessed hand feature map.

[0054] The specific implementation details of step S3 are as follows:

[0055] Step S3 specifically includes the following steps:

[0056] S31 constructs the vertices of the graph, taking key feature areas of the hand such as the fingertips, palm, and joints as the vertices of the graph, and introducing the output results of the pre-trained model as vertex attributes to form a multi-dimensional vertex representation; by introducing the output results of the pre-trained model, each vertex is given multi-dimensional attributes, including shape, position, motion state, etc., which allows the vertices of the graph to not only express the static structure of the hand, but also contain rich dynamic information. This multi-dimensional representation method can more accurately capture the complex changes in gestures, solving the problem that gesture recognition relies heavily on image processing or simple key point detection, which cannot fully capture the details of dynamic changes. Our method can effectively handle complex gestures and subtle changes through multi-dimensional vertex attributes, making recognition in dynamic backgrounds more accurate and robust.

[0057] S32, construct the time layer of the graph, and use the temporally continuous frames as the layers of the graph. Each frame corresponds to a hand graph structure, and the temporal layers are dynamically connected through event streams to realize the transmission of timing information; by organizing the temporally continuous frames into layers of the graph and dynamically connecting them through event streams, the effective transmission of timing information is realized, and the temporal characteristics of gesture evolution are fully captured. This solves the problem of the lack of explicit modeling of timing information in traditional methods, which leads to low recognition accuracy of fast gestures or complex movements, and the inability to accurately model dynamic changes in the time dimension, especially when processing fast and complex gestures, which is prone to distortion. The present invention uses dynamic temporal layer connections to enable each frame to carry complete timing information, thereby improving the system's ability to capture continuous movements and rapid changes.

[0058] S33, constructs spatial structure connections. In the same time frame, the hand feature vertices are connected as edges to construct the spatial structure, and the weights of the edges are dynamically adjusted according to the spatial topological relationship of the gestures to enhance the expressive ability of the spatial structure. By dynamically adjusting the weights of the edges, the relative positions and movements of the various parts of the hand are flexibly expressed according to the spatial topological relationship of the gestures, providing a more accurate spatial structure representation, and solving the problem that existing methods are rarely able to effectively express the relative movements and spatial relationships between the various parts of the hand. The present invention introduces a flexible weight adjustment mechanism in spatial modeling, which can more accurately identify the dynamic changes of the hand in complex gestures, especially in subtle gesture transitions or complex hand movements, and shows extremely high accuracy.

[0059] S34, constructing time associations: In adjacent frames, connect vertices at the same position as edges to construct time associations, and weight the time associations using the timestamp information of the event stream to improve the accuracy of time series modeling; use the timestamp information of the event stream to weight the time associations between each frame, thereby accurately capturing the temporal details of gesture movements. This method can dynamically adjust the temporal relationship based on the timestamp of each frame, avoiding the problem of time errors that traditional methods cannot handle. By weighting time associations, the present invention can accurately capture subtle changes in time, improve the accuracy of time series modeling, and make the system more sensitive and accurate when processing dynamic gestures.

[0060] S35, multimodal fusion and graph convolution, synchronizes the timestamps of the event stream with the RGB frame, generates a time window that matches the RGB frame through dual-channel accumulation of the positive and negative poles of the event stream, dynamically maps the time window to a graph structure stored in the adjacency matrix, and performs a graph convolution operation on the graph structure to extract multi-channel features of multimodal fusion, where each channel contains temporal, spatial, and motion pattern information. This method fuses the multimodal data of the event stream and RGB image, and performs in-depth analysis of the graph structure through graph convolution. This fusion method can fully utilize the high-precision temporal information provided by the event stream and the spatial information provided by the RGB image, effectively improving the comprehensive capabilities of gesture recognition. It solves the difficulty of achieving effective fusion of multimodal data, especially in complex environments, where the combination of RGB images and event streams often leads to information redundancy or mismatch. By combining dynamic mapping of time windows with graph structures, this invention effectively solves the problem that traditional technologies cannot simultaneously process temporal, spatial, and motion pattern information, greatly improving recognition accuracy and adaptability.

[0061] In S31, the adjacency matrix is ​​used to store potential associations in time and space. First, the key feature areas of the hand (such as fingertips, knuckles, palm center, etc.) are used as vertices of the graph.

[0062] In S32, temporally consecutive frames are used as graph layers to ensure that each frame contains a hand graph structure.

[0063] In S33, spatial structural connections are constructed: in the same time frame, connected hand feature vertices are stored as edges (e.g., connections from the palm to each finger).

[0064] In S34, a temporal association is constructed: vertices at the same position in adjacent frames are connected and stored as edges (such as the fingertip in the current frame and the fingertip in the next frame).

[0065] In S35, the timestamp of the event stream is synchronized with the RGB frame, and a time window matching the RGB frame is generated by accumulating the positive and negative channels of the event. This window is dynamically mapped to the graph structure stored in the adjacency matrix, and a graph convolution operation is performed to obtain multi-channel features of multimodal fusion. Each channel contains information such as the temporal and spatial motion pattern.

[0066] The pre-trained model is a deep learning framework that integrates a convolutional neural network (CNN) and a posture regression network. It achieves high-precision regression and recognition of the three-dimensional spatial coordinates and category attributes of hand key points (such as fingertips, joints, and palms) by performing end-to-end training on large-scale public hand key point datasets (such as FreiHAND or Hand-143K). Specifically, the joint points extracted by the pre-trained model from the input RGB image and event stream image are used as vertices of the graph structure. Each vertex is simultaneously assigned multi-dimensional attributes, including: its position vector in three-dimensional space, shape feature descriptors of the current frame (such as the degree of finger opening, the convex hull area of ​​the palm region, etc.), and motion state encoding based on the model output (such as category labels such as static, sliding, and rotation). It can further be expanded to add historical trajectory point features of the vertex in the recent time series to form a vertex representation that integrates multi-dimensional spatiotemporal and semantic fusion, thereby achieving multi-angle perception and expression of the hand state.

[0067] The specific calculation method for each dimension of the multidimensional attributes of the vertex is as follows: the shape attribute is obtained by binary segmentation of the hand area in each frame, combining the Euclidean distance and angle between key points and regional geometric features (such as convex hull area, angle between fingers, palm width, etc.), and further using principal component analysis (PCA) or autoencoder to reduce the dimension to generate a compact shape feature vector; the motion state is automatically distinguished by analyzing the time difference and velocity vector of the vertex spatial position in consecutive frames, such as static, translation, rotation, opening and closing. The features can be described by Δx, Δy, Δz and direction cosines; the pre-trained gesture classification attribute directly calls the gesture recognition model to classify the global gesture type of the current frame or time sequence segment, and the category probability distribution vector is attached to the vertex attribute; the above multidimensional attributes are finally spliced ​​into a high-dimensional feature vector, which comprehensively represents the shape, motion and category characteristics of the vertex.

[0068] The specific synchronization method for the event flow to connect consecutive frames is as follows: First, through the hardware time synchronization mechanism, the time bases of the event camera and the RGB-D camera acquisition module are unified to ensure that the timestamps of the event stream are strictly aligned with the acquisition time points of each RGB frame. Subsequently, within each RGB frame acquisition window, all events in the event stream that overlap with the window time are intercepted and divided into two channels of positive (brightness increase) and negative (brightness decrease) according to the event timestamps, and cumulative statistics are respectively performed to generate positive and negative event count maps corresponding to the RGB frame time window. The algorithm process is as follows: For each frame [t, t + Δt], all events in the event stream E = {ei=(xi, yi, ti, pi)} that satisfy t ≤ ti < t + Δt are counted, and the quantities of two polarities of pi = +1 and pi = -1 are respectively accumulated to form positive and negative event window mapping graphs in the form of W+(x, y) and W-(x, y). This cumulative result further participates in the graph structure construction and multi-modal feature fusion to achieve precise alignment and information enhancement of the event stream and RGB frames in space and time.

[0069] The specific method for the event stream timestamp to adaptively adjust the edge weights is as follows: When constructing the graph structure, for each edge between a pair of vertices vi and vj, its spatial weight ws is calculated by the Gaussian kernel function according to the spatial distance dij between the vertices, ws = exp(-dij2 / 2σs2), where σs is the spatial scale parameter; the time weight wt is adaptively adjusted according to the event stream timestamp interval Δtij, and a weight assignment method similar to temporal decay is adopted, wt = exp(-Δtij / τ), where τ is the time scale hyperparameter; the total weight Wij of the edge is Wij = α·ws + β·wt, where α and β are adjustable weighting coefficients that can be adjusted according to the importance of spatial or temporal correlation; the spatially varying weight automatically increases / decreases the spatial weight by real-time detecting the change in the vertex spatial topology (such as the rapid separation or closing of fingers), and the temporally varying weight dynamically fine-tunes the time weight in combination with the burst intensity of the event stream or the temporal local change trend to enhance the sensitivity of the model to drastic spatial structure changes and temporal dynamic evolutions.

[0070] The specific structure of performing graph convolution operations on the graph structure is to use a two-branch dynamic graph convolutional network to process the adjacency matrix and node features from RGB static features and event stream dynamic features, respectively. Each branch contains multiple layers of spatial graph convolution (such as GCN, GraphSAGE, or GAT structures), which implement cross-channel propagation of spatial and temporal information at each layer, and introduce feature normalization and nonlinear activation after each layer. The multimodal feature fusion mechanism is a parallel interactive attention mechanism and feature splicing. First, the two modal features are weighted by attention fusion, then spliced ​​at the fusion layer and passed to the downstream MLP decoder. The process of mapping the time window to the adjacency matrix is ​​to update the corresponding element Aij(t) in the adjacency matrix A(t) in real time, using the positive and negative event graphs accumulated in each time window as weights. That is, if the event intensity in the window is greater than a threshold, the weight of the related edge is increased, and vice versa. This process is automatically triggered by a gating mechanism or threshold discrimination, realizing the dynamic evolution of the adjacency matrix and the injection of multimodal information in time, ensuring that the graph structure can continuously and adaptively reflect changes in hand state and movement.

[0071] The specific implementation details of step S4 are as follows:

[0072] S41. Construct an encoder. The encoder inputs the spatial information channel of the multimodal fusion feature and uses the additional information contained in the other channels as "additional conditions" to map them together into the latent space. It outputs the parameters of the latent distribution of each joint point, the mean vector μ and the variance vector σ. The resulting latent feature Z = μ + σ·∈, where ∈~N(0,I) is noise sampled from a Gaussian distribution.

[0073] S42. Construct a decoder, perform channel superposition on the feature Z sampled from the latent space and the "additional conditions", and obtain the final predicted hand posture through MLP.

[0074] The implementation details of the deep learning algorithm of this embodiment include the following steps:

[0075] S1. Forward propagation step:

[0076] Input preprocessing: The input RGB image is preliminarily processed through a neural network, and the local transformation parameters of the hand area are dynamically adjusted to ensure that the model can focus on the key areas of the hand.

[0077] Spatiotemporal feature alignment: The event stream and image features are input into a neural network module based on spatiotemporal association. The temporal and spatial relationship of the hand joints is modeled through a graph structure to achieve temporal and spatial alignment of the event stream and image features.

[0078] Joint inference: The fused features are input into the encoder and jointly inferred by the decoder to predict and estimate the 3D position information of the hand node in the world coordinate system.

[0079] S2. Calculate the loss using three loss functions:

[0080] Position loss: The mean squared error (MSE) is used to measure the difference between the predicted hand joint positions and the true positions.

[0081] Reconstruction loss: Use reconstruction loss to ensure that the hand pose predicted by the decoder is as close as possible to the true pose.

[0082] KL divergence loss: KL divergence loss is added to normalize the distribution of the latent space and improve the stability and generalization ability of the generative model.

[0083] S3. Backpropagation optimization:

[0084] Optimizer selection: Adopt an adaptive optimization algorithm, such as the Adam optimizer, to accelerate convergence and improve training efficiency.

[0085] Learning rate scheduling: Dynamically adjust the learning rate according to the training progress and adopt a learning rate decay strategy to prevent the model from falling into the local optimum in the later training.

[0086] Regularization technology: L2 regularization and Dropout layer are introduced to prevent overfitting of the model and improve its generalization ability.

[0087] Batch Normalization: Batch Normalization is used between layers to speed up the training process and stabilize model performance.

[0088] The steps for deep learning model training and deployment are as follows:

[0089] Data collection: Using multi-view RGB-D cameras and an event camera to capture partial gesture data as real data, we adaptively interpolate RGB images and then use an event simulator to generate synthetic event streams as synthetic data. This process then constructs ground truth data to provide data support for deep learning.

[0090] Model training: Make full use of deep learning technology to conduct detailed training on the collected gesture data to build an efficient and accurate hand gesture recognition model;

[0091] Model optimization: Continuously iterate and optimize the model and increase data enhancement to improve the accuracy of gesture intent recognition;

[0092] Model deployment: deploy the trained model to the embedded recognition module, giving it powerful hand gesture recognition capabilities.

[0093] The gesture recognition model described in this embodiment includes a hand recognition module and a recording module. The hand recognition module is responsible for identifying and tracking hand gestures and converting them into a key sequence that is easy for a computer to understand. The recording module is responsible for recording and storing the "password".

[0094] The hand gesture sequence may include but is not limited to common continuous hand movements such as stretching, grasping, digital gestures, pinching gestures, and flipping gestures. These hand gesture sequences can directly control the switch of the target electronic lock module.

[0095] The target electronic locks to which the method of this embodiment is applied include:

[0096] Mechanical structure lock, a switch lock with a mechanical structure based on an embedded chip or intelligent circuit;

[0097] Software application lock. Operating system-based, virtual application lock.

[0098] The electronic device module of this embodiment is characterized by including an infrared ranging sensor, an event camera, an RGB camera, a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. The infrared ranging sensor, event camera, RGB camera, and processor are connected via a bus, and the processor is coupled to the memory. When the processor executes the computer program, the visual multimodal contactless gesture unlocking method is implemented.

[0099] The above-mentioned specific implementation methods are used to illustrate the present invention and are only preferred embodiments of the present invention, rather than limiting the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit of the present invention and the scope of protection of the claims shall fall within the scope of protection of the present invention.

Claims

1. A visual multimodal contactless gesture unlocking method, characterized by: The following steps are involved: S1. Multi-source sensor data acquisition: Using an infrared ranging sensor module to sense approaching hand movements, triggering and scheduling multi-source camera modules to collect hand data, obtaining hand motion data with high dynamic range and rich texture information; S2. Preprocessing of RGB image data: Determine the sampling points of the RGB frame position and color channels using an offset matrix, dynamically adjust the position of each sampling point, and reweight the sampling area to amplify channels that more strongly express hand features, thereby initially extracting hand features. S3. Fusion of event streams and RGB images. Based on preliminary hand feature extraction, the spatial structure of the hand is expressed using finger and palm features as vertices and connecting lines as edges. Temporal associations are constructed by connecting the same vertices in consecutive frames as edges. The event stream is then dynamically mapped onto a graph structure to obtain multimodal fusion features. S4. Multimodal joint prediction: A neural network model that maps multimodal fusion features to a latent space, outputs the parameters μ and σ of the latent distribution, and generates the latent feature Z through reparameterization techniques; After the latent feature Z is superimposed with the additional conditional channel, it is decoded by the multi-layer perceptron MLP to obtain the final hand posture prediction; In step S3, the joint points extracted by the pre-trained hand posture estimation model are used as vertices, and the multi-dimensional attributes of shape, position and motion state are fused. The continuous frames are dynamically connected through the event stream to construct a temporal association. The gesture evolution process is explicitly modeled using a hierarchical graph structure, and the edge weights are adaptively adjusted using the event stream timestamps. Through the dynamic spatial topology weighting and timestamp weighting mechanisms, the hand structure representation and motion trend capture are simultaneously optimized. Combined with the dual-branch graph convolutional network, the static features of the RGB image and the high-precision dynamic information of the event stream are deeply integrated.

2. The visual multimodal contactless gesture unlocking method according to claim 1, characterized in that: In step S1, during data collection, multiple-view RGB-D cameras and an event camera are used to capture partial gesture data as real data. RGB images are first adaptively interpolated and then a synthetic event stream is generated as synthetic data through an event simulator to construct the true value.

3. The visual multimodal contactless gesture unlocking method according to claim 1, wherein: Step S2 specifically includes the following steps: S21, through a lightweight neural network to predict the sampling offset of each pixel position, generate an offset tensor of shape ijk2, the offset dynamically adjusts the sampling position of the convolution kernel to focus on the key area of ​​the hand, uses bilinear interpolation to calculate the pixel value of the offset point, and performs conventional convolution on the offset sampling area to obtain feature one. S22, split the feature one by channel, calculate the global average of each channel, generate the weight of each channel through the activation function, use the learnable parameters to reweight the channels of the original feature map, enhance the channels expressing the hand features, suppress background noise, and obtain the preprocessed hand feature map.

4. The visual multimodal contactless gesture unlocking method according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31, construct the vertices of the graph, taking the key feature areas of the hand such as fingertips, palms, joints, etc. as the vertices of the graph, and introducing the output results of the pre-trained model as vertex attributes to form a multi-dimensional vertex representation; S32, constructing the time layer of the graph, using temporally consecutive frames as the graph layer. Each frame corresponds to a hand graph structure. The time layers are dynamically connected through event streams to achieve the transmission of temporal information. S33, building spatial structure connections, connects hand feature vertices as edges to build spatial structures in the same time frame, and dynamically adjusts the weights of edges according to the spatial topological relationship of gestures to enhance the expressiveness of spatial structures; S34, build temporal associations: In adjacent frames, connect vertices at the same position as edges to build temporal associations. The temporal associations are weighted by the timestamp information of the event stream to improve the modeling accuracy of the time series. S35, multimodal fusion and graph convolution, synchronizes the timestamp of the event stream with the RGB frame, generates a time window that matches the RGB frame through the positive and negative dual-channel accumulation of the event stream, dynamically maps the time window to the graph structure stored in the adjacency matrix, performs graph convolution operations on the graph structure, and extracts multi-channel features of multimodal fusion. Each channel contains time, space, and motion pattern information.

5. The visual multimodal contactless gesture unlocking method according to claim 4, characterized in that: The pre-trained model is a deep learning framework that integrates a convolutional neural network (CNN) and a posture regression network. It achieves high-precision regression and recognition of the three-dimensional spatial coordinates and category attributes of the hand key points by performing end-to-end training on a large-scale public hand key point dataset: specifically, the joint points extracted by the pre-trained model for the input RGB image and event stream image are used as vertices of the graph structure, and each vertex is simultaneously assigned multi-dimensional attributes, including: its position vector in three-dimensional space, the shape feature descriptor of the current frame, and the motion state encoding based on the model output. It can be further expanded to add historical trajectory point features of the vertex in the most recent time series to form a vertex representation that integrates multi-dimensional spatiotemporal and semantic fusion, thereby achieving multi-angle perception and expression of the hand state. The posture regression network is the HRNet or MediaPipe Hands hand posture estimation model.

6. The visual multimodal contactless gesture unlocking method according to claim 5, characterized in that: The specific calculation method for each dimension of the multidimensional attributes of a vertex is as follows: the shape attribute is obtained by binary segmentation of the hand area in each frame, combining the Euclidean distance, angle, and regional geometric features between key points, and further using principal component analysis or autoencoder dimensionality reduction to generate a compact shape feature vector; the motion state is automatically distinguished by analyzing the time difference and velocity vector of the vertex spatial position in consecutive frames, such as stillness, translation, rotation, and opening and closing; the features can be described by Δx, Δy, Δz, and direction cosines; the pre-trained gesture classification attribute directly calls the gesture recognition model to classify the global gesture type of the current frame or time sequence segment, and the category probability distribution vector is attached to the vertex attribute; the above multidimensional attributes are finally spliced ​​into a high-dimensional feature vector to comprehensively characterize the shape, motion, and category characteristics of the vertex.

7. The visual multimodal contactless gesture unlocking method according to claim 1, characterized in that: The specific synchronization method of dynamically connecting consecutive frames of the event stream is as follows: first, the time base of the event camera and the RGB-D camera acquisition module are unified through the hardware time synchronization mechanism to ensure that the timestamp of the event stream is strictly aligned with the acquisition time point of each RGB frame; Subsequently, within each RGB frame acquisition window, all events in the event stream that overlap with the window in time are intercepted, divided into two channels of positive and negative poles according to the event timestamps, and cumulatively statistically processed respectively to generate positive and negative event count maps corresponding to the RGB frame time window. The algorithm process is as follows: for each frame [t, t+Δt], count all events in the event stream E={e i =(x i , y i , t i , p i )} that satisfy t ≤ t i < t+Δt, and cumulatively count the quantities of the two polarities of p i =+1 and p i =-1 respectively, to form a positive and negative event window mapping diagram in the form of W + (x, y), W - (x, y); this cumulative result further participates in the construction of the graph structure and multi-modal feature fusion to achieve precise alignment and information enhancement of the event stream and the RGB frame in space and time.

8. The visual multimodal contactless gesture unlocking method according to claim 1, characterized in that: The specific method of adaptively adjusting the edge weight of the event stream timestamp is as follows: when constructing the graph structure, each pair of vertices v i With v j The edge between them has a spatial weight w s The spatial distance d between vertices ij Calculated by Gaussian kernel function, w s =exp(-d ij 2 / 2σ s 2 ), where σ s is the spatial scale parameter; the time weight w t According to the event stream timestamp interval Δt ij Adaptive adjustment, using a weight distribution method similar to time series decay, w t =exp(-Δt ij / τ), τ is the time scale hyperparameter; the total weight of the edge W ij =α·w s +β·w t , where α and β are adjustable weighting coefficients that can be adjusted according to the importance of spatial or temporal associations; spatially variable weights automatically increase / decrease spatial weights by detecting vertex spatial topology changes in real time, and temporally variable weights dynamically fine-tune temporal weights based on the burst intensity of event flows or local temporal change trends, thereby improving the model's sensitivity to sudden changes in spatial structure and temporal dynamic evolution.

9. The visual multimodal contactless gesture unlocking method according to claim 4, characterized in that: The specific structure of performing the graph convolution operation on the graph structure is to use a dual-branch dynamic graph convolution network to process the adjacency matrix and node features from the RGB static features and the event flow dynamic features respectively; Each branch contains multiple layers of spatial graph convolution, which realizes cross-channel propagation of spatial and temporal information at each layer, and introduces feature normalization and nonlinear activation after each layer; The multimodal feature fusion mechanism is a parallel interactive attention mechanism and feature splicing. First, the two modal features are weightedly fused, then spliced ​​in the fusion layer and passed to the downstream MLP decoder. The process of mapping the time window to the adjacency matrix is ​​as follows: the positive and negative event graphs of the event flow accumulated in each time window are used as weights, and the corresponding elements A in the adjacency matrix A(t) are updated in real time. ij (t), that is, if the event intensity within the window is greater than the threshold, the weight of the related edge is enhanced, otherwise it is weakened; this process is automatically triggered by the gating mechanism or threshold discrimination, realizing the dynamic evolution of the adjacency matrix in time series and the injection of multimodal information, ensuring that the graph structure can continuously and adaptively reflect changes in hand status and movement.

10. The visual multimodal contactless gesture unlocking method according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41, the composition of the encoder, the spatial information channel and additional conditions of the multimodal fusion feature are used as encoder inputs and mapped together to the latent space, outputting the parameters of the potential distribution of each joint point, the mean vector μ and the variance vector σ, and obtaining the latent feature Z=μ+σ·∈, ∈~N(0,I) is the noise sampled from the Gaussian distribution, where the additional conditions are the extra information contained in other channels besides the spatial information channel of the multimodal fusion feature; S42, construct a decoder, superimpose the latent feature Z and the additional conditions on each channel, obtain the concatenated feature shape, and decode it through the multi-layer perceptron MLP to obtain the final predicted hand posture.

Images