Gesture recognition methods, devices and storage media

Abstract

This application provides a gesture recognition method, device, and storage medium. The method includes: acquiring multiple depth image data of a gesture to be recognized; wherein the multiple depth image data originate from different acquisition perspectives; performing a first preprocessing on the depth image data to obtain corresponding first preprocessed point cloud data; performing point cloud fusion processing on the first preprocessed point cloud data to obtain fused point cloud data; performing a second preprocessing on the fused point cloud data to obtain second preprocessed point cloud data; and performing feature extraction and feature fusion processing on the second preprocessed point cloud data to obtain the recognition result of the gesture to be recognized. This achieves low-latency, high-precision gesture recognition.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a gesture recognition method, device and storage medium. Background Technology

[0002] With the development of virtual reality and augmented reality technologies, gesture capture, as an important human-computer interaction technology, has been widely used in film and television production, industrial control, medical training and other fields.

[0003] Currently, dynamic gesture capture mainly falls into three categories: wearable sensing, optical vision capture, and marker-based motion capture.

[0004] However, these technical solutions are easily affected by external factors, making it difficult to simultaneously meet the requirements of low latency, high accuracy, low equipment cost, high reliability, and high portability in gesture recognition. Summary of the Invention

[0005] This application provides a gesture recognition method, device, and storage medium to solve the problems that related technologies are easily affected by external factors, resulting in poor timeliness and low recognition accuracy.

[0006] In a first aspect, this application provides a gesture recognition method, including:

[0007] Acquire multiple depth image data of the gesture to be recognized; wherein the multiple depth image data are obtained from different acquisition perspectives;

[0008] The depth image data is subjected to a first preprocessing to obtain the corresponding first preprocessed point cloud data;

[0009] Based on the first preprocessed point cloud data, point cloud fusion processing is performed to obtain fused point cloud data;

[0010] The fused point cloud data is subjected to a second preprocessing to obtain the second preprocessed point cloud data;

[0011] Based on the second preprocessed point cloud data, feature extraction and feature fusion are performed to obtain the recognition result of the gesture to be recognized.

[0012] In one possible implementation, acquiring multiple depth image data of the gesture to be recognized includes: acquiring multiple depth image data of the gesture to be recognized through multiple preset sensors; wherein the multiple preset sensors are used to cover a preset space around the gesture to be recognized.

[0013] In one possible implementation, the preset sensor includes a first preset sensor, a second preset sensor, and a third preset sensor; correspondingly, the step of acquiring multiple depth image data of the gesture to be recognized through multiple preset sensors includes: acquiring top view depth image data of the gesture to be recognized through the first preset sensor; acquiring lower left view depth image data of the gesture to be recognized through the second preset sensor; and acquiring lower right view depth image data of the gesture to be recognized through the third preset sensor.

[0014] In one possible implementation, the step of performing feature extraction and feature fusion processing on the second preprocessed point cloud data to obtain the recognition result of the gesture to be recognized includes: acquiring a trained point cloud deep learning network and a trained fully connected fusion network; wherein the trained point cloud deep learning network is trained using the second preprocessed point cloud data samples and corresponding point cloud feature labels, and the trained fully connected fusion network is trained using the point cloud feature labels and corresponding feature space labels; performing feature extraction and feature fusion processing on the second preprocessed point cloud data based on the trained point cloud deep learning network and the trained fully connected fusion network to obtain the feature space of the gesture to be recognized; and generating the recognition result of the gesture to be recognized based on the feature space.

[0015] In one possible implementation, before obtaining the trained point cloud deep learning network and the trained fully connected fusion network, the method further includes: obtaining an initial point cloud deep learning network and an initial fully connected fusion network; configuring a loss function based on dynamic weights for the initial point cloud deep learning network and the initial fully connected fusion network; and training the initial point cloud deep learning network and the initial fully connected fusion network using the second preprocessed point cloud data samples, the point cloud feature labels, and the feature space labels, according to the loss function based on dynamic weights, to obtain the trained point cloud deep learning network and the trained fully connected fusion network.

[0016] In one possible implementation, the first preprocessing of the depth image data to obtain corresponding first preprocessed point cloud data includes: performing dimensional coordinate transformation processing on the depth image data to obtain corresponding three-dimensional point cloud data; performing coordinate system unification processing on the three-dimensional point cloud data to obtain corresponding unified point cloud data; and performing temporal alignment processing based on the unified point cloud data to obtain the first preprocessed point cloud data.

[0017] In one possible implementation, the step of performing point cloud fusion processing based on the first preprocessed point cloud data to obtain fused point cloud data includes: performing outlier filtering processing on the first preprocessed point cloud data to obtain filtered point cloud data; and performing point cloud fusion processing based on the filtered point cloud data using an adaptive fusion strategy of least squares to obtain the fused point cloud data.

[0018] In one possible implementation, the second preprocessing of the fused point cloud data to obtain second preprocessed point cloud data includes: normalizing and rigidly transforming the fused point cloud data to obtain the second preprocessed point cloud data.

[0019] Secondly, this application provides a gesture recognition device, including: a memory and a processor;

[0020] The memory stores computer-executed instructions;

[0021] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0022] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0023] The gesture recognition method, device, and storage medium provided in this application achieve comprehensive acquisition of gesture data by using depth image data from different acquisition perspectives, overcoming the unavoidable hand self-occlusion problem in traditional single-view acquisition. Through point cloud fusion processing, the complete three-dimensional structural information of the hand can be accurately reconstructed. Combined with feature extraction and feature fusion processing, the recognition result of the gesture to be recognized is obtained. This achieves low-latency, high-precision gesture recognition. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] Figure 1 This is a schematic diagram illustrating an application scenario of the gesture recognition method provided in the embodiments of this application;

[0026] Figure 2 A flowchart illustrating a gesture recognition method provided in one embodiment of this application;

[0027] Figure 3This is a schematic diagram of a preset sensor spatial distribution provided in one embodiment of this application;

[0028] Figure 4 A schematic diagram of a gesture recognition system provided in an embodiment of this application;

[0029] Figure 5 A schematic diagram of the fused point cloud data and gesture recognition results provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of the gesture recognition device provided in the embodiments of this application;

[0031] Figure 7 This is a schematic diagram of the structure of the gesture recognition device provided in an embodiment of this application.

[0032] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0034] Optical vision capture solutions employ depth cameras or binocular vision systems to achieve non-contact gesture recognition through 3D reconstruction algorithms. This involves integrating a depth camera or binocular camera into a virtual reality headset to capture hand images and using deep learning algorithms for keypoint detection and dynamic hand modeling. However, optical vision capture suffers from limitations in field of view and blind spots due to single-view imaging, making it prone to hand occlusion and resulting in gesture recognition failures or inaccuracies. Furthermore, real-time computation requires significant computing resources, increasing hardware costs and causing signal processing latency and reduced device battery life. Additionally, this solution is highly sensitive to ambient lighting conditions and susceptible to external interference, leading to low gesture recognition accuracy.

[0035] The gesture recognition method provided in this application addresses the problems of poor timeliness and low accuracy in related technologies. By acquiring depth image data from different perspectives, it achieves comprehensive acquisition of gesture data, overcoming the unavoidable hand self-occlusion problem in traditional single-view acquisition. Then, through point cloud fusion processing, it can accurately reconstruct the complete three-dimensional structural information of the hand. Combined with feature extraction and feature fusion processing, the recognition result of the gesture to be recognized is obtained. This achieves low-latency, high-precision gesture recognition.

[0036] Figure 1 This is a schematic diagram illustrating an application scenario of the gesture recognition method provided in the embodiments of this application, such as... Figure 1 As shown, the scene is a computer device, including: a receiving device 101, a processor 102 and a display device 103.

[0037] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the item recognition method. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0038] In the specific implementation process, the receiving device 101 can be an input / output interface or a communication interface, and can acquire multiple depth image data of the gesture to be recognized.

[0039] The processor 102 can process multiple depth image data of the gesture to be recognized in order to determine the recognition result of the gesture.

[0040] The display device 103 can be used to display the recognition results of the above-mentioned gestures to be recognized.

[0041] The display device can also be a touch screen, used to receive user commands while displaying the above content, so as to achieve interaction with the user.

[0042] It should be understood that the aforementioned processor can be implemented by reading instructions from memory and executing those instructions, or it can be implemented through chip circuitry.

[0043] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0044] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0045] Figure 2 This is a flowchart illustrating a gesture recognition method provided in one embodiment of this application. The execution entity of this embodiment can be... Figure 1 The computer equipment shown is not specifically limited in this embodiment. Figure 2 As shown, the method includes:

[0046] S201: Acquire multiple depth image data of the gesture to be recognized; wherein, the multiple depth image data come from different acquisition perspectives.

[0047] Specifically, multiple depth image data of the gesture to be recognized are collected through multiple preset sensors; wherein, the multiple preset sensors are evenly distributed in a ring in space to cover the preset space around the gesture to be recognized.

[0048] The preset space can completely cover all areas of the gesture to be recognized.

[0049] The preset sensors include a first preset sensor, a second preset sensor, and a third preset sensor.

[0050] The first, second, and third preset sensors are fixedly connected to the wristband via a connecting device, and the wristband is worn on the wrist. The first, second, and third preset sensors are evenly distributed in a ring shape in space and are kept relatively fixed to the forearm and hand by the connecting device. Figure 3 As shown, one preset sensor is located directly above the hand to acquire top-down view information such as palm opening and closing, and fingertip movements; the other two preset sensors are positioned at preset angles on the lower left and right sides of the hand, perpendicular to the wrist's central axis, to collect key information that is difficult for the top sensor to capture, such as finger bending. The preset angle can be 30 degrees.

[0051] By employing an optimized layout strategy that involves multiple preset sensors evenly distributed in a ring in space, the inherent hand self-occlusion problem in traditional single-view acquisition is overcome, ensuring the integrity and accuracy of hand motion data.

[0052] The three preset sensors form a triangular data acquisition network.

[0053] By forming a triangular acquisition network with three preset sensors, multi-dimensional synchronous acquisition of hand movement information is achieved. The point cloud data acquired by multiple sensors are highly complementary and can reconstruct the complete three-dimensional structural information of the hand.

[0054] It should be noted that this application does not limit the number and location of the preset sensors, and the number and location of the preset sensors can be flexibly adjusted according to the accuracy requirements of the actual application scenario.

[0055] The circular, uniformly distributed structure of multiple pre-defined sensors in this application exhibits excellent fault tolerance. Even if individual sensors are temporarily obstructed or subjected to environmental interference, the entire system can still maintain stable operation by relying on data from other sensors, ensuring the continuity and reliability of gesture recognition. It provides a flexible approach to performance optimization, enabling customized deployments for different application scenarios. By increasing the number of sensors, more detailed hand movement information can be obtained, thereby further improving the accuracy and reliability of gesture recognition.

[0056] Specifically, the first preset sensor acquires top-view depth image data of the gesture to be recognized. The second preset sensor acquires bottom-left-view depth image data of the gesture. The third preset sensor acquires bottom-right-view depth image data of the gesture.

[0057] The preset sensor can be a lidar sensor.

[0058] S202: Perform a first preprocessing on the depth image data to obtain the corresponding first preprocessed point cloud data.

[0059] Specifically, S202 includes S2021 to S2023:

[0060] S2021: Perform dimensional coordinate transformation on the depth image data to obtain the corresponding 3D point cloud data.

[0061] Specifically, the depth image data is transformed into corresponding 3D point cloud data by using the depth map conversion module, based on the mapping of the preset sensor intrinsic parameter matrix and perspective projection.

[0062] Wherein, the point cloud corresponding to the k-th sensor in the sensor coordinate system The coordinates of each 3D point cloud data are:

[0063]

[0064] In the formula, The image represents the depth image acquired by the k-th sensor (k = 1, 2, 3) at time t, with dimensions h × w; (u, v) represents the coordinates of a pixel in the depth image; θh θ represents the horizontal field of view of the k-th sensor; v This represents the vertical field of view of the k-th sensor.

[0065] S2022: Perform coordinate system unification processing on the 3D point cloud data to obtain the corresponding unified point cloud data.

[0066] Specifically, there are three preset sensors, located at the top, lower left, and lower right of the hand, respectively. A unified world coordinate system is established with the top sensor (k=1) as the reference sensor, and the point cloud data of the other sensors are mapped into this coordinate system through rigid body transformation.

[0067] The formula for calculating rigid transformation is as follows:

[0068]

[0069] In the formula, This represents the point cloud coordinates transformed from the k-th sensor coordinate system to the reference sensor coordinate system. This represents the point cloud coordinates collected by the k-th sensor at time t, with the coordinate system being the k-th sensor's own coordinate system; This represents the rotation matrix from the k-th sensor coordinate system k to the reference sensor coordinate system 1, satisfying orthogonality. It represents a translation vector that describes the position of the origin of coordinate system k in coordinate system 1; This represents a column vector of all 1s, used to broadcast a translation vector to all points in the point cloud data.

[0070] S2023: Based on the unified point cloud data, perform time-series alignment processing to obtain the first preprocessed point cloud data.

[0071] Specifically, interpolation is performed based on timestamps and motion estimation to align unified post-point cloud data from different times to a preset reference time, in order to obtain the first pre-processed post-point cloud data.

[0072] Optionally, an independent data cache queue can be configured for each preset sensor, and the latest point cloud data can be extracted from the data cache queue in real time.

[0073] Quasi-synchronous sampling through time window constraints can ensure the consistency of point cloud data in the time dimension, laying the foundation for subsequent spatial fusion.

[0074] S203: Based on the preprocessed point cloud data, perform point cloud fusion processing to obtain fused point cloud data.

[0075] Specifically, S203 includes S2031 to S2032:

[0076] S2031: Perform outlier filtering on the first preprocessed point cloud data to obtain filtered point cloud data.

[0077] Specifically, for each point in the first preprocessed point cloud data, the average distance from that point to its nearest neighbors is calculated to obtain the average distance of all points; based on the average distance of all points, the statistical characteristics of the average distance are calculated; based on the statistical characteristics, a filtering threshold is obtained; if the average distance from a point to its nearest neighbors is greater than the filtering threshold, then that point is filtered.

[0078] The formula for calculating the average distance from a point to its nearest neighbors is:

[0079]

[0080] In the formula, p i This represents the three-dimensional coordinates of the i-th point in the point cloud data after the first preprocessing. d represents the set of k nearest neighbors of point pi; i p i arrive average distance; p j Represents the set of nearest neighbors The j-th point in the array.

[0081] The formula for calculating the average distance between all points is:

[0082]

[0083] In the formula, N represents the total number of points in the point cloud data; μ represents the global average distance, that is, the average distance d from all points to their k nearest neighbors. i The mean.

[0084] The formula for calculating statistical characteristics is as follows:

[0085]

[0086] In the formula, σ represents the standard deviation of the distance.

[0087] The formula for calculating the filtering threshold is as follows:

[0088] a=μ+ασ

[0089] In the formula, a represents the filtering threshold; α represents the threshold parameter.

[0090] S2032: Based on the filtered point cloud data, perform point cloud fusion processing using an adaptive fusion strategy based on least squares to obtain fused point cloud data.

[0091] Specifically, an observation equation is established; the covariance matrix is ​​determined based on the error characteristics of each sensor; the minimum sum of squared errors is obtained based on the covariance matrix and the observation equation; and the optimal fusion estimate is obtained based on the minimum sum of squared errors to obtain the fused point cloud data.

[0092] The formula for the observation equation is:

[0093] z k =H k x+v k k = 1, 2, 3

[0094] In the formula, z k Let H be the observation value of the k-th sensor, x be the actual spatial location, and H be the distance between the observations. k Let v be the observation matrix of the k-th sensor. k To observe noise.

[0095] The formula for minimizing the sum of squared errors is as follows:

[0096]

[0097] In the formula, J(x) represents minimizing the sum of squared errors; This represents the inverse of the covariance matrix of the k-th sensor.

[0098] The formula for calculating the optimal fusion estimate is as follows:

[0099]

[0100] In the formula, P t This represents the fused point cloud data.

[0101] S204: Perform a second preprocessing on the fused point cloud data to obtain the second preprocessed point cloud data.

[0102] Specifically, the fused point cloud data is normalized and subjected to rigid transformation to obtain the second preprocessed point cloud data.

[0103] Specifically, the maximum and minimum values ​​of the fused point cloud data are calculated in each coordinate axis direction; normalization is performed based on the maximum and minimum values ​​to obtain normalized point cloud data; and rigid transformation is performed on the normalized point cloud data to obtain the second preprocessed point cloud data.

[0104] The formulas for calculating the maximum and minimum values ​​are as follows:

[0105] x max =max i＝1,…,N x i ,x min =min i＝1,…,Nx i ,y max =max i＝1,…,N y i

[0106] y min =min i＝1,…,N y i ,z max =max i＝1,…,N z i ,z min =min i＝1,…,N z i

[0107] In the formula, x max x min These represent the maximum and minimum values ​​of the fused point cloud data along the x-axis, respectively; y max y min These represent the maximum and minimum values ​​of the fused point cloud data along the y-axis, respectively; z max z min These represent the maximum and minimum values ​​of the fused point cloud data along the z-axis, respectively; x i y i z i These represent the coordinate values ​​of the fused point cloud data in each coordinate axis direction; N represents the total number of points in the point cloud data.

[0108] Among them, the normalized point cloud Each point in It can be represented as:

[0109]

[0110] The calculation formula for rigid transformation processing of normalized point cloud data is as follows:

[0111]

[0112] In the formula, X t This represents the point cloud data after the second preprocessing: For rotation matrix, It is a translation vector. R is a vector consisting entirely of 1s. t and t t By analyzing point clouds Principal component analysis was performed.

[0113] S205: Based on the second preprocessed point cloud data, perform feature extraction and feature fusion processing to obtain the recognition result of the gesture to be recognized.

[0114] Specifically, S205 includes S2051 to S2053:

[0115] S2051: Obtain the trained point cloud deep learning network and the trained fully connected fusion network; wherein, the trained point cloud deep learning network is trained using the preprocessed point cloud data samples and the corresponding point cloud feature labels, and the trained fully connected fusion network is trained using the point cloud feature labels and the corresponding feature space labels.

[0116] S2052: Based on the trained point cloud deep learning network and the trained fully connected fusion network, feature extraction and feature fusion processing are performed on the second preprocessed point cloud data to obtain the feature space of the gesture to be recognized.

[0117] Optionally, feature extraction processing is performed on the second preprocessed point cloud data to obtain the point cloud features at the current time; the gesture recognition result at the previous time is obtained; a hot start mechanism is used to perform data preprocessing on the point cloud features at the current time and the gesture recognition result at the previous time to obtain temporal context features; and feature fusion processing is performed on the temporal context features to obtain the feature space of the gesture to be recognized.

[0118] The calculation formula for feature extraction is as follows:

[0119]

[0120] In the formula, This represents a feature extraction network; Represents the parameters of the network; z t This represents the extracted point cloud features at time t.

[0121] The formula for calculating the temporal context features is as follows:

[0122]

[0123] In the formula, h t For the temporal context features containing historical information, ψ is the feature construction parameter for the warm-start mechanism. This is the gesture recognition result from the previous moment.

[0124] The calculation formula for feature fusion processing is as follows:

[0125]

[0126] In the formula, f represents the feature space of the gesture to be recognized at time t; θ θ represents a fully connected converged network; θ represents the network parameters.

[0127] By using feature fusion based on temporal information, the continuity and accuracy of gesture recognition are improved.

[0128] S2053: Generate the recognition result of the gesture to be recognized based on the feature space.

[0129] Specifically, an inverse transformation is performed on the feature space to generate the recognition result of the gesture to be recognized.

[0130] The formula for the inverse transformation of the feature space is as follows:

[0131]

[0132] In the formula, The inverse transformation function representing a coordinate transformation; This represents the inverse of the rotation matrix.

[0133] The gesture recognition method provided in this application achieves comprehensive acquisition of gesture data by using depth image data from different acquisition perspectives, overcoming the unavoidable hand self-occlusion problem in traditional single-view acquisition. Through point cloud fusion processing, it can accurately reconstruct the complete three-dimensional structural information of the hand. Combined with feature extraction and feature fusion processing, the recognition result of the gesture to be recognized is obtained. This achieves low-latency, high-precision gesture recognition.

[0134] In one embodiment of this application, based on the above embodiment, before step S2051, a process of training a point cloud deep learning network and a fully connected fusion network is further included, as detailed below:

[0135] S2054: Obtain the initial point cloud deep learning network and the initial fully connected fusion network.

[0136] S2055: Configure a loss function based on dynamic weights for the initial point cloud deep learning network and the initial fully connected fusion network.

[0137] The formula for the loss function based on dynamic weights is as follows:

[0138]

[0139] In the formula, f θ Represents a fully connected fusion network; ψ is the feature construction parameter of the hot-start mechanism; This represents a feature extraction network; T represents the truth value of the gesture at time t; ||·|| represents a norm metric in the gesture space; the superscript (i) indicates that the sample comes from the i-th sample sequence in the training set; T i The length of the i-th sample sequence is represented by N; N represents the total number of points in the point cloud data. It is an adaptive function based on the magnitude of gesture changes at adjacent time points. The input is two n-dimensional vectors, and the output is a real number; n represents the dimension of the gesture output vector.

[0140] It should be noted that when a drastic change in gesture is detected (i.e., and If the difference is significant, the weight of that sample will be automatically increased.

[0141] S2056: Based on the loss function based on dynamic weights, the initial point cloud deep learning network and the initial fully connected fusion network are trained using the second preprocessed point cloud data samples, point cloud feature labels, and feature space labels to obtain the trained point cloud deep learning network and the trained fully connected fusion network.

[0142] The gesture recognition method provided in this application improves the recognition accuracy of fast gestures and enhances the recognition performance of high-dynamic gestures by combining a loss function based on dynamic weights during the training process of the point cloud deep learning network and the fully connected fusion network.

[0143] Figure 4 This is a schematic diagram of a gesture recognition system provided in an embodiment of this application. Figure 4 As shown, the gesture recognition system includes a LiDAR sensor, a depth map conversion module, a point cloud fusion module, a gesture recognition module, and an adaptive gesture dynamic weight adjustment module.

[0144] The lidar sensors, used to acquire depth image data and send it to the depth conversion module, include a top lidar sensor, a lower left lidar sensor, and a lower right lidar sensor. The top lidar sensor acquires depth image data from the top viewpoint. The lower left lidar sensor acquires depth image data from the lower left viewpoint. The lower right lidar sensor acquires depth image data from the lower right viewpoint.

[0145] The depth map conversion module is used to convert depth image data from different acquisition perspectives into point cloud data from different acquisition perspectives, and then send the point cloud data from different acquisition perspectives to the point cloud fusion module.

[0146] The point cloud fusion module unifies and fuses point cloud data from different acquisition perspectives to obtain fused point cloud data, which is then sent to the gesture recognition module. This module includes: a time synchronization module, a point cloud registration module, a coordinate transformation module, and a noise filtering and fusion module. The time synchronization module synchronizes the point cloud data from different acquisition perspectives. The point cloud registration module registers the coordinate systems of the time-synchronized point cloud data from different acquisition perspectives. The coordinate transformation module transforms the coordinate-registered point cloud data from different acquisition perspectives to a reference coordinate system. The noise filtering and fusion module identifies and removes outliers in the unified coordinate system point cloud data and fuses the point cloud data from different acquisition perspectives.

[0147] The gesture recognition module performs gesture recognition based on the fused point cloud data at the current moment and the preliminary gesture recognition result from the previous moment, obtaining a preliminary gesture recognition result, and then sends this preliminary result to the adaptive gesture dynamic weight adjustment module. It includes a preprocessing module, a feature extraction module, a warm-start module, and a fully connected network. The preprocessing module performs normalization and rigid transformation on the fused point cloud data at the current moment; sends the processed point cloud data to the feature extraction module; and acquires and sends the preliminary gesture recognition result from the previous moment to the warm-start module. The feature extraction module extracts features from the processed point cloud data at the current moment, obtaining the point cloud features for the current moment, and sends these features to the warm-start module. The warm-start module preprocesses the preliminary gesture recognition result from the previous moment and the point cloud features at the current moment to obtain temporal context features, and sends these temporal context features to the fully connected network. The fully connected network performs feature fusion processing on the temporal context features to obtain a preliminary gesture recognition result, and then sends this preliminary result to the adaptive gesture dynamic weight adjustment module and the preprocessing module.

[0148] The adaptive gesture dynamic weight adjustment module is used to dynamically adjust the weights of the initial gesture recognition results to obtain the final gesture recognition result.

[0149] Figure 5 This is a schematic diagram of the fused point cloud data and gesture recognition results provided in an embodiment of this application.

[0150] The gesture recognition method provided in this application achieves comprehensive acquisition of gesture data by using depth image data from different acquisition perspectives, overcoming the unavoidable hand self-occlusion problem in traditional single-view acquisition. Through point cloud fusion processing, it can accurately reconstruct the complete three-dimensional structural information of the hand. Combined with feature extraction and feature fusion processing, the recognition result of the gesture to be recognized is obtained. This achieves low-latency, high-precision gesture recognition.

[0151] Figure 6 This is a schematic diagram of the structure of the gesture recognition device provided in the embodiments of this application, as shown below. Figure 6 As shown, the gesture recognition device 60 provided in this embodiment includes: an acquisition module 601, a processing module 602, a fusion module 603, and a recognition module 604.

[0152] The acquisition module 601 is used to acquire multiple depth image data of the gesture to be recognized; wherein the multiple depth image data are from different acquisition perspectives;

[0153] The processing module 602 is used to perform a first preprocessing on the depth image data to obtain corresponding first preprocessed point cloud data;

[0154] The fusion module 603 is used to perform point cloud fusion processing based on the first preprocessed point cloud data to obtain fused point cloud data.

[0155] The processing module 602 is further configured to perform a second preprocessing on the fused point cloud data to obtain second preprocessed point cloud data.

[0156] The recognition module 604 is used to perform feature extraction and feature fusion processing based on the second preprocessed point cloud data to obtain the recognition result of the gesture to be recognized.

[0157] In one possible implementation, the acquisition module 601 is specifically used to: acquire multiple depth image data of the gesture to be recognized through multiple preset sensors; wherein the multiple preset sensors are used to cover a preset space around the gesture to be recognized.

[0158] In one possible implementation, the preset sensor includes a first preset sensor, a second preset sensor, and a third preset sensor; correspondingly, the acquisition module 601 is specifically used to: acquire top view depth image data of the gesture to be recognized through the first preset sensor; acquire lower left view depth image data of the gesture to be recognized through the second preset sensor; and acquire lower right view depth image data of the gesture to be recognized through the third preset sensor.

[0159] In one possible implementation, the recognition module 604 is specifically used for: acquiring a trained point cloud deep learning network and a trained fully connected fusion network; wherein the trained point cloud deep learning network is trained using second preprocessed point cloud data samples and corresponding point cloud feature labels, and the trained fully connected fusion network is trained using the point cloud feature labels and corresponding feature space labels; based on the trained point cloud deep learning network and the trained fully connected fusion network, performing feature extraction and feature fusion processing on the second preprocessed point cloud data to obtain the feature space of the gesture to be recognized; and generating a recognition result of the gesture to be recognized based on the feature space.

[0160] In one possible implementation, the gesture recognition device 60 further includes:

[0161] The training module is used to acquire an initial point cloud deep learning network and an initial fully connected fusion network; configure a loss function based on dynamic weights for the initial point cloud deep learning network and the initial fully connected fusion network; and train the initial point cloud deep learning network and the initial fully connected fusion network using the second preprocessed point cloud data samples, the point cloud feature labels, and the feature space labels according to the loss function based on dynamic weights, so as to obtain the trained point cloud deep learning network and the trained fully connected fusion network.

[0162] In one possible implementation, the processing module 602 is specifically used to: perform dimensional coordinate transformation processing on the depth image data to obtain corresponding three-dimensional point cloud data; perform coordinate system unification processing on the three-dimensional point cloud data to obtain corresponding unified point cloud data; and perform temporal alignment processing based on the unified point cloud data to obtain the first preprocessed point cloud data.

[0163] In one possible implementation, the fusion module 603 is specifically used to: perform outlier filtering on the first preprocessed point cloud data to obtain filtered point cloud data; and perform point cloud fusion processing based on the filtered point cloud data and an adaptive fusion strategy using least squares to obtain the fused point cloud data.

[0164] In one possible implementation, the processing module 602 is specifically used to: perform normalization and rigid transformation processing on the fused point cloud data to obtain the second preprocessed point cloud data.

[0165] The gesture recognition device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0166] Figure 7 This is a schematic diagram of the structure of a gesture recognition device provided in an embodiment of this application. Figure 7 As shown, the gesture recognition device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0167] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0168] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0169] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0170] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0171] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0172] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0173] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0174] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0175] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0176] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0179] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0181] Finally, it should be noted that other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and alterations may be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A gesture recognition method, characterized in that, include: Multiple depth image data of the gesture to be recognized are acquired through multiple preset sensors; wherein, the multiple preset sensors are used to cover a preset space around the gesture to be recognized, and the multiple depth image data are from different acquisition perspectives; The depth image data is subjected to a first preprocessing to obtain the first preprocessed point cloud data corresponding to each depth image data. Based on the first preprocessed point cloud data corresponding to the multiple depth image data, point cloud fusion processing is performed to obtain fused point cloud data. The fused point cloud data is subjected to a second preprocessing to obtain second preprocessed point cloud data, including: normalizing the fused point cloud data and performing rigid transformation processing to obtain second preprocessed point cloud data. Based on the preprocessed point cloud data, feature extraction and feature fusion are performed to obtain the recognition result of the gesture to be recognized. This includes: acquiring a trained point cloud deep learning network and a trained fully connected fusion network; wherein the trained point cloud deep learning network is trained using the preprocessed point cloud data samples and corresponding point cloud feature labels, and the trained fully connected fusion network is trained using the point cloud feature labels and corresponding feature space labels; performing feature extraction on the preprocessed point cloud data to obtain the point cloud features at the current time; acquiring the gesture recognition result from the previous time step; using a warm-start mechanism to preprocess the point cloud features at the current time step and the gesture recognition result from the previous time step to obtain temporal context features; performing feature fusion on the temporal context features to obtain the feature space of the gesture to be recognized; and generating the recognition result of the gesture to be recognized based on the feature space.

2. The method according to claim 1, characterized in that, The preset sensors include a first preset sensor, a second preset sensor, and a third preset sensor; Accordingly, the process of acquiring multiple depth image data of the gesture to be recognized through multiple preset sensors includes: The top-view depth image data of the gesture to be recognized is acquired through the first preset sensor; The lower left perspective depth image data of the gesture to be recognized is acquired through the second preset sensor. The third preset sensor acquires depth image data of the lower right view of the gesture to be recognized.

3. The method according to claim 1, characterized in that, Before obtaining the trained point cloud deep learning network and the trained fully connected fusion network, the following steps are also included: Obtain the initial point cloud deep learning network and the initial fully connected fusion network; For the initial point cloud deep learning network and the initial fully connected fusion network, configure a loss function based on dynamic weights; Based on the loss function based on dynamic weights, the initial point cloud deep learning network and the initial fully connected fusion network are trained using the second preprocessed point cloud data samples, the point cloud feature labels, and the feature space labels, so as to obtain the trained point cloud deep learning network and the trained fully connected fusion network.

4. The method according to claim 1 or 2, characterized in that, The first preprocessing of the depth image data to obtain preprocessed point cloud data corresponding to each depth image data includes: The depth image data is subjected to dimensional coordinate transformation to obtain the three-dimensional point cloud data corresponding to each depth image data. The three-dimensional point cloud data is processed using a coordinate system to obtain the corresponding unified point cloud data; Based on the unified point cloud data, temporal alignment processing is performed to obtain the first preprocessed point cloud data.

5. The method according to claim 1 or 2, characterized in that, The step of performing point cloud fusion processing on the first preprocessed point cloud data corresponding to the multiple depth image data respectively to obtain fused point cloud data includes: Outlier filtering is performed on the first preprocessed point cloud data corresponding to the multiple depth image data respectively to obtain filtered point cloud data. Based on the filtered point cloud data, point cloud fusion processing is performed using an adaptive fusion strategy based on least squares to obtain the fused point cloud data.

6. A gesture recognition device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

Images