A human-computer interaction method and interactive system based on gesture recognition

Abstract

The present invention relates to the field of human-computer interaction technology, and more specifically, to a human-computer interaction method based on gesture recognition and an interaction system thereof, which comprises the following steps: using an RGB camera and a depth sensor to capture two-dimensional image data and depth data of a gesture, and constructing point cloud data based on the two-dimensional image data and the depth data of the gesture; using a key point detection algorithm based on a hand node model to identify and track the key nodes of the hand and each finger on the point cloud data; extracting the dynamic gesture features of the gesture based on the coordinates of the finger nodes and the time series changes; using a convolutional neural network to classify the dynamic gesture features; defining the mapping rules between gestures and machine control commands, and the machine control commands are passed to the corresponding application to trigger the corresponding functions. By introducing spatial smoothness constraints and temporal continuity constraints, the registration process of multi-frame point cloud data is optimized to reduce the phenomenon of jumps or breaks.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a human-computer interaction method based on gesture recognition and an interaction system thereof. Background Art

[0002] The human-computer interaction method based on gesture recognition is an interaction mode that uses advanced computer vision technology and artificial intelligence algorithms to recognize human gestures and communicate and control electronic devices or systems accordingly; the method allows users to convey instructions through intuitive body movements, especially hand and finger movements, without physical contact such as keyboards, mice or other traditional input devices.

[0003] Among them, when pursuing high-speed response and high accuracy through gesture recognition technology, and recognizing complex gestures in real time in a dynamic environment, when the gesture is temporarily blocked, abnormal data points will be introduced, causing abnormal recognition results, and when the gesture changes rapidly, jumping or breaking phenomena will occur, resulting in errors in the matching of point cloud data between consecutive frames, reducing the accuracy of gesture recognition. Therefore, a human-computer interaction method based on gesture recognition and an interaction system thereof are provided. Summary of the invention

[0004] The purpose of the present invention is to provide a human-computer interaction method based on gesture recognition and an interaction system thereof, so as to solve the problem proposed in the above-mentioned background technology that when complex gestures are recognized in real time in a dynamic environment, when the gesture is temporarily blocked, abnormal data points will be introduced, causing abnormal recognition results, and when the gesture changes rapidly, jumping or breaking phenomena will occur, resulting in errors in the matching of point cloud data between consecutive frames, thereby reducing the accuracy of gesture recognition.

[0005] To achieve the above object, the present invention provides a human-computer interaction method based on gesture recognition, comprising the following steps:

[0006] S1: Use RGB camera and depth sensor as data acquisition devices to capture the two-dimensional image data and depth data of gestures, and construct point cloud data based on the two-dimensional image data and depth data of gestures;

[0007] S2: Use the key point detection algorithm based on the hand node model to identify and track the key nodes of the hand and each finger on the point cloud data. Use the iterative closest point algorithm to optimize the posture of the model hand. Consider different viewing angles or time periods to obtain multi-frame point cloud data for registration. Register the nodes of the hand node model with the continuous multi-frame point cloud data to capture the coordinates of the finger nodes.

[0008] The specific steps involved are:

[0009] Use the iterative closest point algorithm (ICP) to find the optimal rotation matrix R and translation vector T:

[0010]

[0011] Where M = {m1,m2,…,m M}; C = {c1, c2, …, c C};

[0012] Where M represents the point set of the hand node model; C represents the point set of the point cloud data; NN(m,C) represents the point in the point cloud data set closest to the model node m; ||·|| represents the Euclidean distance; through iterative solution, in each iteration, the new rotation and translation parameters are calculated after matching the nearest point until convergence, minimizing the distance between the model point set M and the point cloud data point set C;

[0013] For continuous multi-frame point cloud data, the iterative closest point algorithm (ICP) is continuously executed on the time series. By introducing spatial smoothness constraints and temporal continuity constraints, the registration process of multi-frame point cloud data is optimized. The expression of the alignment of multi-frame point cloud data is:

[0014]

[0015] In the formula, {R t ,T t} represents the hand node model posture (rotation and translation) of the tth frame; R t represents the rotation matrix of the hand model relative to the world coordinate system in the frame with timestamp t; T is the total number of frames, which means the number of frames considered in the whole sequence, from the first frame to the last frame; T t represents the translation vector of the hand model relative to the world coordinate system in the frame with timestamp t; P n represents the nth point in the model point cloud; N is an index variable, indicating the total number of points in the point cloud; n is the index of the point; ρ represents a robust kernel function (such as the Huber function), which is used to reduce the influence of outliers; Π represents a projection function, which is used to project a 3D point onto the image plane; represents the observation of the nth model point in the tth frame; temporal represents the weight factor controlling the temporal continuity; spatial A weight factor representing the smoothness of motion; represents the norm of the rotation matrix on the special Euclidean group SO(3), ensuring that the optimization of the rotation matrix is ​​geometrically meaningful;

[0016] S3: Extract the dynamic gesture features of the gesture based on the finger node coordinates and time series changes, such as the change rate of joint angles and the change of distance between fingers, construct the gesture contour through point cloud data, and extract the shape feature vector;

[0017] The specific steps involved in extracting the dynamic gesture features of a gesture are:

[0018] S3.1: Based on the three-dimensional coordinates of the finger nodes in the continuous frames obtained in step S2, the coordinate sequence changes dynamically with the gesture to form time series data;

[0019] S3.2: For each finger, calculate the angles between adjacent joint nodes and analyze the rate of change of these angles over time;

[0020] S3.3: Capture the speed and pattern of finger opening or closing by analyzing the distance between different finger nodes over time;

[0021] For the fingertip nodes of the Ith finger and the kth finger, they are denoted as P INI,t and P kNk,t , N I is the fingertip node index of finger i; N k is the fingertip node index of finger k; then the distance D between the fingertip nodes of the Ith finger and the kth finger at time t Ik,t for:

[0022] D Ik,t =|P INI,t ―P kNk,t |;

[0023] Then the distance change rate between finger nodes ΔD ik,t for:

[0024] ΔD Ik,t =D Ik,t ―D Ik,t―1 ;

[0025] Considering that recent data may better reflect the current gesture dynamics than early data, a higher weight is given to recent data. At the same time, in order to reduce the instantaneous error caused by sensor noise or short-term occlusion, the importance of recent data is emphasized by introducing a weighted moving average, and the distance change rate ΔD is calculated. Ik,t To perform smoothing:

[0026]

[0027] in:

[0028] w(τ―t)=e ―λ(τ―t) ;

[0029] In the formula, w(τ―t) represents the weight function; λ represents the positive decay factor, which is used to control the rate at which the weight decreases over time; ΔT is the size of the time window considered; τ represents the time when the event in the time series occurs; and t represents the current time point.

[0030] Weighted moving average smoothes data by assigning different weights to data at different time points in the time series. Recent data points are given higher weights, which better represent the current dynamic trend, while farther data points are given lower weights.

[0031] S4: Use convolutional neural networks to classify dynamic gesture features. The model matches the input features with the learned patterns and receives real-time gesture data to output the category of the user's current gesture.

[0032] S5: Define the mapping rules between gestures and machine control commands, for example, clenching a fist means confirmation, opening a palm means cancellation, etc.; the machine control command is passed to the corresponding application to trigger the corresponding function.

[0033] As a further improvement of the technical solution, in S1, the specific steps of constructing the point cloud are:

[0034] S1.1: The RGB camera captures the two-dimensional image information of the gesture, i.e., the pixel values ​​of the three color channels of red (R), green (G), and blue (B), to form a color image;

[0035] S1.2: The depth sensor synchronously records the depth information of the gesture and generates a depth map, in which each pixel represents the distance from the point to the sensor;

[0036] S1.3: Calibrate the internal and external parameters of the RGB camera and depth sensor to ensure that the coordinate systems of the two are aligned so that the color information and depth information of each pixel can be accurately matched, and the RGB image and the depth map are aligned. A calibration board (such as a chessboard) is used to perform a joint calibration of the internal and external parameters and the external parameters. Using the calibration results, the pixel coordinate systems of the RGB image and the depth map are aligned through geometric transformation (rotation and translation) so that each RGB pixel can find its corresponding depth value.

[0037] S1.4: Using the intrinsic parameters of the RGB camera (focal length f x ,f y , main point c x ,c y ) and the depth value of each pixel point, converting the two-dimensional pixel coordinates into three-dimensional coordinates;

[0038] S1.5: Add RGB color value to each 3D coordinate point to form point cloud data with color information.

[0039] As a further improvement of the technical solution, in S1.4, the specific steps of converting the three-dimensional coordinates are:

[0040] Convert the two-dimensional pixel coordinates (u, v) to normalized image plane coordinates (xn ,y n ):

[0041]

[0042] In the formula, f x represents the focal length of the camera along the width of the image (horizontally); f y represents the focal length of the camera along the image height (vertical direction); (c x ,c y ) represents the image center coordinates; (x n ,y n ) represents the coordinates in a coordinate system with the camera optical center as the origin and parallel to the image plane; u represents the horizontal coordinate of the pixel; v represents the vertical coordinate of the pixel;

[0043] For each normalized coordinate (x n ,y n ), using the depth value Z corresponding to the pixel point (i.e. the distance from the point to the camera), calculate the actual coordinates (X, Y, Z) of the point in three-dimensional space:

[0044] X=Z·x n ; Y = Z·y n ;

[0045] And transform the above three-dimensional coordinates from the camera coordinate system to the world coordinate system through rotation and translation transformation:

[0046]

[0047] In the formula, Represents the coordinates in the camera coordinate system; Represents the coordinates after conversion to the world coordinate system; R represents the rotation matrix; T represents the translation vector.

[0048] As a further improvement of the technical solution, the key point detection algorithm is based on the hand node model, and the specific identification steps for the key nodes are:

[0049] The point cloud is divided into different regions by using the K-means clustering algorithm, at least one of which should contain the hand point cloud; the actual hand point cloud data of the current frame acquired by the depth camera is filtered and denoised by using a mean filter to remove isolated points and noise points;

[0050] Predefine the hand node model including the basic structure of wrist, palm, fingers and their key node layout, and align its initial posture with the hand area in the point cloud data;

[0051] Based on the preliminary alignment, the iterative closest point algorithm (ICP) is used to iteratively minimize the error projected from the hand node model to the point cloud, and continuously adjust the posture of the hand node model until the best fit is achieved;

[0052] Using the preset key node positions in the model, the aligned model is compared with the actual point cloud data, and the coordinates of the key nodes in the real three-dimensional space are located through the nearest neighbor search algorithm.

[0053] As a further improvement of the technical solution, the hand node model is specifically:

[0054] The hand node model has N key nodes, each of which can be represented by a three-dimensional coordinate (X i ,Y i ,Z i ), where i = 1, 2, ..., N, the hand node model includes the following nodes:

[0055] Wrist node: P1 = (X wrist ,Y wrist ,Z wrist );

[0056] Palm center node: P2 = (X palmC ,Y palmC ,Z palmC );

[0057] For each finger, there are three joint nodes and a fingertip node:

[0058] Among them, j = 1, 2, 3, 4, 5; k = 1, 2, 3, 4;

[0059] For the bending angle between any adjacent joints, the joint angle is usually used to describe the bending degree of the finger. The bending angle θ between the nth joint and the n+1th joint is calculated. n , where node P n Represents the position of the nth joint, node P n+1 represents the position of the n+1th joint, then:

[0060]

[0061] In the formula, Represents the vector from the nth joint node to the n+1th joint node, reflecting the relative position between the two joints; Represents the vector from the nth joint node to the wrist node, which serves as a reference for calculating the angle to ensure that the calculation of the joint bending angle has a consistent direction; · represents the vector dot product; |·| represents the modulus of the vector.

[0062] As a further improvement of the technical solution, the gesture contour is constructed through point cloud data and the shape feature vector is extracted, which involves the following steps:

[0063] The convex hull algorithm is used to construct the gesture contour through the point cloud data, and the boundary points representing the gesture contour are screened out from the point cloud;

[0064] The shape features of the gesture are represented by the Histogram of Directed Gradients (HOG). For the Histogram of Directed Gradients (HOG), the steps are:

[0065] Segment the gesture contour into small cell units;

[0066] The orientation histogram of pixel gradients is calculated within each cell to capture the local edge orientation distribution;

[0067] Adjacent cells are combined into blocks, and the histogram of oriented gradients (HOG) features of each block are calculated, which are further aggregated into a global descriptor;

[0068] In order to capture the shape changes of dynamic gestures, the gesture shape feature vectors of each frame are concatenated to form a time series feature vector.

[0069] As a further improvement of the technical solution, the specific steps of the convolutional neural network for classifying dynamic gesture features are:

[0070] The dynamic gesture features are normalized to ensure that different features have the same scale. The sliding window technique is used to continuously capture the changes in gestures. Each segment is used as an input sample of the convolutional neural network model.

[0071] Construct a convolutional neural network model and train it using labeled gesture samples and dynamic gesture features, so that the model can learn the gesture features corresponding to each gesture sample.

[0072] Deploy the trained convolutional neural network model, which receives and processes real-time gesture data and outputs the corresponding gesture category.

[0073] On the other hand, the present invention provides a human-computer interaction system based on gesture recognition, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the human-computer interaction methods based on gesture recognition described above.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. In the human-computer interaction method based on gesture recognition and its interactive system, when recognizing complex gestures in real time in a dynamic environment, recent data is given a higher weight, taking into account that it may be more able to reflect the current gesture dynamics than early data, and at the same time, to reduce the instantaneous error caused by sensor noise or short-term occlusion; by introducing weighted moving average, when the gesture is temporarily occluded, abnormal data points will be introduced. The weighted moving average considers the average of multiple data points, which can reduce the impact of such abnormal values ​​to a certain extent and maintain the consistency of the recognition results.

[0076] 2. In the human-computer interaction method based on gesture recognition and its interactive system, for continuous multi-frame point cloud data, the iterative closest point algorithm (ICP) is continuously executed in the time series, and the registration process of multi-frame point cloud data is optimized by introducing spatial smoothness constraints and temporal continuity constraints. Continuous execution of ICP can establish an accurate correspondence between point cloud data at different time points, and can maintain the continuity of tracking even when the gesture changes rapidly, reducing jumps or breaks;

[0077] By introducing spatial smoothness constraints, local abnormal matches caused by sensor noise, environmental interference or incomplete point cloud can be filtered out during the registration process, making the gesture model smoother and more accurate, thereby improving the robustness of recognition;

[0078] Introducing time continuity constraints helps maintain consistency between sequences and prevents small registration errors from accumulating over time in long sequence data processing, causing serious deviations in the final results, thereby ensuring the long-term accuracy of gesture recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0080] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0081] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0082] Embodiment 1:

[0083] See also Figure 1 As shown, this embodiment provides a human-computer interaction method based on gesture recognition, comprising the following steps:

[0084] S1. Use RGB camera and depth sensor as data acquisition equipment to capture the two-dimensional image data and depth data of gestures, and construct point cloud data based on the two-dimensional image data and the depth data of gestures. Compared with relying only on two-dimensional image information, point cloud with depth information can significantly improve the accuracy and robustness of gesture recognition. Depth information can help distinguish foreground and background, reduce misrecognition caused by lighting, complex background, etc., and make gesture tracking and recognition more accurate.

[0085] In this embodiment, the specific steps of constructing the point cloud are:

[0086] S1.1: The RGB camera captures the two-dimensional image information of the gesture, i.e., the pixel values ​​of the three color channels of red (R), green (G), and blue (B), to form a color image;

[0087] S1.2: The depth sensor synchronously records the depth information of the gesture and generates a depth map, in which each pixel represents the distance from the point to the sensor;

[0088] S1.3: Calibrate the internal and external parameters of the RGB camera and depth sensor to ensure that the coordinate systems of the two are aligned so that the color information and depth information of each pixel can be accurately matched, and the RGB image and the depth map are aligned. A calibration board (such as a chessboard) is used to perform a joint calibration of the internal and external parameters and the external parameters. Using the calibration results, the pixel coordinate systems of the RGB image and the depth map are aligned through geometric transformation (rotation and translation) so that each RGB pixel can find its corresponding depth value.

[0089] S1.4: Using the intrinsic parameters of the RGB camera (focal length f x ,f y , main point c x ,c y) and the depth value of each pixel point, converting the two-dimensional pixel coordinates into three-dimensional coordinates;

[0090] In this embodiment, the specific steps of converting the three-dimensional coordinates are:

[0091] Convert the two-dimensional pixel coordinates (u, v) to normalized image plane coordinates (x n ,y n ):

[0092]

[0093] In the formula, f x represents the focal length of the camera along the width of the image (horizontally); f y represents the focal length of the camera along the image height (vertical direction); (c x ,c y ) represents the image center coordinates; (x n ,y n ) represents the coordinates in a coordinate system with the camera optical center as the origin and parallel to the image plane; u represents the horizontal coordinate of the pixel; v represents the vertical coordinate of the pixel;

[0094] For each normalized coordinate (x n ,y n ), using the depth value Z corresponding to the pixel point (i.e. the distance from the point to the camera), calculate the actual coordinates (X, Y, Z) of the point in three-dimensional space:

[0095] X=Z·x n ; Y = Z·y n ;

[0096] And transform the above three-dimensional coordinates from the camera coordinate system to the world coordinate system through rotation and translation transformation:

[0097]

[0098] In the formula, Represents the coordinates in the camera coordinate system; Represents the coordinates after conversion to the world coordinate system; R represents the rotation matrix; T represents the translation vector.

[0099] S1.5: Add RGB color value to each 3D coordinate point to form point cloud data with color information.

[0100] S2: A key point detection algorithm based on the hand node model is used to identify and track the key nodes of the hand and each finger on the point cloud data. The iterative closest point algorithm (ICP) is used to optimize the posture of the model hand. Multi-frame point cloud data is obtained from different viewing angles or time periods for registration. The nodes of the hand node model are registered with the continuous multi-frame point cloud data to capture the coordinates of the finger nodes.

[0101] Specifically, the key point detection algorithm is based on the hand node model, and the specific identification steps for the key nodes are:

[0102] The point cloud is divided into different regions by using the K-means clustering algorithm, at least one of which should contain the hand point cloud; the actual hand point cloud data of the current frame acquired by the depth camera is filtered and denoised by using a mean filter to remove isolated points and noise points;

[0103] Predefine the hand node model including the basic structure of wrist, palm, fingers and their key node layout, and align its initial posture with the hand area in the point cloud data;

[0104] Based on the preliminary alignment, the iterative closest point algorithm (ICP) is used to iteratively minimize the error projected from the hand node model to the point cloud, and continuously adjust the posture of the hand node model until the best fit is achieved;

[0105] Using the preset key node positions in the model, the aligned model is compared with the actual point cloud data, and the coordinates of the key nodes in the real three-dimensional space are located through the nearest neighbor search algorithm.

[0106] In this embodiment, the hand node model is specifically:

[0107] The hand node model has N key nodes, each of which can be represented by a three-dimensional coordinate (X i ,Y i ,Z i ), where i = 1, 2, ..., N, the hand node model includes the following nodes:

[0108] Wrist node: P1 = (X wrist ,Y wrist ,Z wrist );

[0109] Palm center node: P2 = (X palmC ,Y palmC ,Z palmC );

[0110] For each finger, there are three joint nodes and a fingertip node:

[0111] Among them, j = 1, 2, 3, 4, 5; k = 1, 2, 3, 4;

[0112] For the bending angle between any adjacent joints, the joint angle is usually used to describe the bending degree of the finger. The bending angle θ between the nth joint and the n+1th joint is calculated. n , where node P n Represents the position of the nth joint, node P n+1 represents the position of the n+1th joint, then:

[0113]

[0114] In the formula, Represents the vector from the nth joint node to the n+1th joint node, reflecting the relative position between the two joints; Represents the vector from the nth joint node to the wrist node, which is used as a reference for calculating the angle to ensure that the calculation of the joint bending angle has a consistent direction; · represents the vector dot product; |·| represents the modulus of the vector;

[0115] Specifically, if node 2 is the root node of the index finger (MP joint) and node 3 is the first joint node of the index finger, then its bending angle

[0116]

[0117] In the formula, Represents the vector from the root node of the index finger to the first joint node; Represents the reference vector from the index finger root node along the palm direction.

[0118] In this embodiment, the nodes of the hand node model are aligned with the continuous multi-frame point cloud data, and the specific steps involved are:

[0119] Use the iterative closest point algorithm (ICP) to find the optimal rotation matrix R and translation vector T:

[0120]

[0121] Where M = {m1,m2,…,m M}; C = {c1, c2, …, c C};

[0122] Where M represents the point set of the hand node model; C represents the point set of the point cloud data; NN(m,C) represents the point in the point cloud data set closest to the model node m; ||·|| represents the Euclidean distance; through iterative solution, in each iteration, the new rotation and translation parameters are calculated after matching the nearest point until convergence, minimizing the distance between the model point set M and the point cloud data point set C;

[0123] For continuous multi-frame point cloud data, the iterative closest point algorithm (ICP) is continuously executed on the time series. By introducing spatial smoothness constraints and temporal continuity constraints, the registration process of multi-frame point cloud data is optimized. The expression of the alignment of multi-frame point cloud data is:

[0124]

[0125] In the formula, {R t ,T t} represents the hand node model posture (rotation and translation) of the tth frame; R t represents the rotation matrix of the hand model relative to the world coordinate system in the frame with timestamp t; T is the total number of frames, which means the number of frames considered in the whole sequence, from the first frame to the last frame; T t represents the translation vector of the hand model relative to the world coordinate system in the frame with timestamp t; P n represents the nth point in the model point cloud; N is an index variable, indicating the total number of points in the point cloud; n is the index of the point; ρ represents a robust kernel function (such as the Huber function), which is used to reduce the influence of outliers; Π represents a projection function, which is used to project a 3D point onto the image plane; represents the observation of the nth model point in the tth frame; temporal represents the weight factor controlling the temporal continuity; spatial A weight factor representing the smoothness of motion; represents the norm of the rotation matrix on the special Euclidean group SO(3), ensuring that the optimization of the rotation matrix is ​​geometrically meaningful;

[0126] By limiting the change range of the rotation matrix R and translation vector T between adjacent frames through time continuity constraints, it helps to eliminate the drastic jumps caused by noise or local optimality, making the registration results of the entire sequence smoother and more coherent, ensuring the natural transition of gestures and avoiding unreasonable mutations;

[0127] Spatial smoothness constraints help maintain the continuity and rationality of the model in space, especially when dealing with complex gestures or fast movements. They can reduce the overall deviation caused by local matching errors, improve the robustness of the system, and help the model better adapt to complex changes in the real environment, such as incomplete point cloud data caused by light changes and occlusions.

[0128] When processing continuous actions, by forcing the model's posture to be continuous in time and smoothly change in space, the dynamic changes of the hand nodes can be tracked more accurately, especially in fast or subtle movements. This constraint can significantly improve the accuracy of node positioning and reduce drift.

[0129] Specifically, the registration process of multi-frame point cloud data is optimized by iterative closest point algorithm (ICP) and introducing spatial smoothness constraints and temporal continuity constraints. The optimized rotation matrix R t and the translation vector T t , adjust the posture of the hand node model to the state that best matches the current frame point cloud data; make each node coordinate in the model (joint nodes and fingertip nodes of each finger) by adjusting the model coordinate according to R t Rotation and translation T t To map to the actual 3D space position of the current frame, after mapping the model nodes to the actual space coordinates, they need to be matched with the point cloud data of the current frame to identify and track the key nodes and capture the coordinates of the finger nodes.

[0130] S3: Extract the dynamic gesture features of the gesture based on the finger node coordinates and time series changes, such as the change rate of joint angles and the change of distance between fingers, construct the gesture contour through point cloud data, and extract the shape feature vector;

[0131] In this embodiment, the specific steps involved in extracting the dynamic gesture features of the gesture are:

[0132] S3.1: Based on the three-dimensional coordinates of the finger nodes in the continuous frames obtained in step S2, the coordinate sequence changes dynamically with the gesture to form time series data;

[0133] For the i-th finger, there are N i There are key nodes (including joints and fingertips). At the tth moment, the three-dimensional coordinates of the jth node can be expressed as (X ij,t ,Y ij,t ,Z ij,t ), where i = 1, 2, ..., 5 (corresponding to five fingers), j = 1, 2, ..., N i , and t = 1, 2, ..., T, T represents the total number of frames; therefore, the time series data set for the entire gesture can be represented as a high-dimensional array, where each element corresponds to the position of a specific finger node at a time point:

[0134] The time series dataset of the entire gesture is represented as:

[0135] D={(P ij,t )|i=1,2,…,5; j=1,2,…,N i ; t=1,2,…,T};

[0136] Where P ij,t =(X ij,t ,Y ij,t ,Z ij,t ) is the coordinate vector of the jth node of the ith finger at time point t;

[0137] S3.2: For each finger, calculate the angles between adjacent joint nodes and analyze the rate of change of these angles over time; this involves calculating the difference in joint angles between adjacent frames to reflect the finger bending speed and pattern;

[0138] The rate of change of joint angle is:

[0139] Δθ n,t =θ n,t ―θ n,t―1 ;

[0140] In the formula, θ n,t represents the bending angle of the nth joint in the tth frame; θ n,t―1 represents the bending angle of the nth joint in the t-1th frame;

[0141] S3.3: By analyzing the distance changes between different finger nodes over time, the speed and pattern of finger opening or closing are captured, which helps to recognize gestures such as grasping and stretching;

[0142] For the fingertip nodes of the Ith finger and the kth finger, they are denoted as P INI,t and P kNk,t , N I is the fingertip node index of finger i; N k is the fingertip node index of finger k; then the distance D between the fingertip nodes of the Ith finger and the kth finger at time t Ik,t for:

[0143] D Ik,t =|P INI,t ―P kNk,t |;

[0144] Then the distance change rate between finger nodes ΔD ik,t for:

[0145] ΔD Ik,t =D Ik,t ―D Ik,t―1 ;

[0146] Considering that recent data may better reflect the current gesture dynamics than early data, a higher weight is given to recent data. At the same time, in order to reduce the instantaneous error caused by sensor noise or short-term occlusion, the importance of recent data is emphasized by introducing a weighted moving average, and the distance change rate ΔD is calculated. IK,t To perform smoothing:

[0147]

[0148] in:

[0149] w(τ―t)=e ―λ(τ―t) ;

[0150] In the formula, w(τ―t) represents the weight function; λ represents the positive decay factor, which is used to control the rate at which the weight decreases over time; ΔT is the size of the time window considered; τ represents the time when the event in the time series occurs; and t represents the current time point.

[0151] Weighted moving average smoothes data by assigning different weights to data at different time points in the time series. Recent data points are given higher weights, which better represent the current dynamic trend, while farther data points are given lower weights.

[0152] Smoothing data is used to reduce the impact of noise, and through a weighted mechanism, the model is made more sensitive to changes in recent data, thereby improving the tracking accuracy of dynamic gesture features.

[0153] Furthermore, the gesture contour is constructed through the point cloud data, and the shape feature vector is extracted. The shape features, such as the length, width ratio, and curvature of the contour, are extracted to form a feature vector, which specifically involves the following steps:

[0154] The convex hull algorithm is used to construct the gesture contour through the point cloud data, and the boundary points representing the gesture contour are screened out from the point cloud;

[0155] The shape features of the gesture are represented by the Histogram of Directed Gradients (HOG). For the Histogram of Directed Gradients (HOG), the steps are:

[0156] Segment the gesture contour into small cell units;

[0157] The orientation histogram of pixel gradients is calculated within each cell to capture the local edge orientation distribution;

[0158] Adjacent cells are combined into blocks, and the histogram of oriented gradients (HOG) features of each block are calculated, which are further aggregated into a global descriptor;

[0159] In gesture recognition, the Histogram of Oriented Gradients (HOG) feature can effectively capture the edge information and shape features of the gesture contour, and provide a stable and reliable feature representation even when the lighting conditions change or the background is complex. By calculating the Histogram of Oriented Gradients (HOG) features of gestures in consecutive frames and analyzing the changes of these features over time (through time series analysis), the recognition ability of dynamic changes of gestures is further enhanced. The gesture contour is constructed through the point cloud and the shape feature vector is extracted. In order to more effectively capture and express the characteristics of gestures, the accuracy and robustness of the recognition system are improved.

[0160] In order to capture the shape changes of dynamic gestures, the gesture shape feature vectors of each frame are concatenated to form a time series feature vector.

[0161] S4: Use convolutional neural networks to classify dynamic gesture features. The model matches the input features with the learned patterns and receives real-time gesture data to output the category of the user's current gesture.

[0162] The specific steps of the convolutional neural network to classify dynamic gesture features are:

[0163] The dynamic gesture features are normalized to ensure that different features have the same scale. The sliding window technique is used to continuously capture the changes in gestures. Each segment is used as an input sample of the convolutional neural network model.

[0164] Construct a convolutional neural network model and train it using labeled gesture samples and dynamic gesture features, so that the model can learn the gesture features corresponding to each gesture sample.

[0165] Constructing a convolutional neural network:

[0166] Input layer: The size of the input layer is determined according to the dimension of the feature vector and the length of the time series;

[0167] Convolutional layers: Design multiple convolutional layers to extract spatiotemporal features, and use 2D / 3D convolutions to capture both spatial and temporal information when processing image sequences;

[0168] A pooling layer is added after the convolutional layer to reduce the feature dimension, reduce the amount of calculation, and keep important features;

[0169] Recurrent structure: Considering the temporal dependency of gesture sequences, a recurrent neural network (such as LSTM or GRU units) can be connected after the convolutional layer to specifically handle the long-term dependency problem in time series data;

[0170] Fully connected layer: After feature extraction, one or more fully connected layers are used to integrate the features, which are used to integrate the learned features and map them to category outputs;

[0171] Classification layer: The final fully connected layer, whose number of neurons corresponds to the number of predefined gesture categories, cooperates with the softmax activation function to output the probability distribution of each gesture category;

[0172] Initialization parameters: Assign initial values ​​to the weights and bias items in the network through Xavier;

[0173] Define the loss function: Select the cross entropy loss function to measure the gap between the predicted output and the actual label;

[0174] Choose an optimizer: Configure a variant of gradient descent through the Adam optimizer to update the network weights to minimize the loss function;

[0175] The trained convolutional neural network model is deployed, which receives and processes real-time gesture data and outputs corresponding gesture categories. The mapping from physical gestures to predefined commands is realized, thereby controlling the application or device to perform corresponding operations, wherein the real-time gesture data includes dynamic gesture features.

[0176] S5: Define the mapping rules between gestures and machine control commands, for example, clenching a fist means confirmation, opening a palm means cancellation, etc.; the machine control command is passed to the corresponding application to trigger the corresponding function.

[0177] Embodiment 2:

[0178] This embodiment provides a human-computer interaction system based on gesture recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the human-computer interaction methods based on gesture recognition described above.

[0179] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A human-computer interaction method based on gesture recognition, characterized in that: The interactive method comprises the following steps: S1: Use an RGB camera and a depth sensor to capture the two-dimensional image data and depth data of the gesture, and construct point cloud data based on the two-dimensional image data and the depth data of the gesture; S2: Use the key point detection algorithm based on the hand node model to identify and track the key nodes of the hand and each finger on the point cloud data, use the iterative closest point algorithm to optimize the posture of the model hand, and align the nodes of the hand node model with the continuous multi-frame point cloud data to capture the finger node coordinates. The specific steps involved are: Use the iterative closest point algorithm to find the optimal rotation matrix R and translation vector T: Where M = {m1,m2,…,m M }; C = {c1, c2, …, c C }; Where M represents the point set of the hand node model; C represents the point set of the point cloud data; NN(m,C) represents the point in the point cloud data set closest to the model node m; ||·|| represents the Euclidean distance; For continuous multi-frame point cloud data, the iterative closest point algorithm is continuously executed on the time series. By introducing spatial smoothness constraints and temporal continuity constraints, the registration process of multi-frame point cloud data is optimized. The expression of the alignment of multi-frame point cloud data is: In the formula, {R t ,T t } represents the hand node model posture of the tth frame; R t represents the rotation matrix of the hand model relative to the world coordinate system in the frame with timestamp t; T is the total number of frames, which means the number of frames considered in the whole sequence, from the first frame to the last frame; T t represents the translation vector of the hand model relative to the world coordinate system in the frame with timestamp t; P n represents the nth point in the model point cloud; N is the index variable, representing the total number of points in the point cloud; n is the index of the point; ρ represents the robust kernel function; Π represents the projection function; represents the observation of the nth model point in the tth frame; temporal represents the weight factor controlling the temporal continuity; spatial A weight factor representing the smoothness of motion; represents the norm of the rotation matrix on the special Euclidean group SO(3); S3: Extract the dynamic gesture features of the gesture based on the finger node coordinates and time series changes, construct the gesture contour through the point cloud data, and extract the shape feature vector; The specific steps of extracting the dynamic gesture features of a gesture are: S3.1: Based on the three-dimensional coordinates of the finger nodes in the continuous frames obtained in step S2, the coordinate sequence changes dynamically with the gesture to form time series data; S3.2: For each finger, calculate the angles between adjacent joint nodes and analyze the rate of change of these angles over time; S3.3: Capture the speed and pattern of finger opening or closing by analyzing the distance between different finger nodes over time; For the fingertip nodes of the Ith finger and the kth finger, they are denoted as P INI,t and P kNk,t , N I is the fingertip node index of finger i; N k is the fingertip node index of finger k; then the distance D between the fingertip nodes of the Ith finger and the kth finger at time t Ik,t for: D Ik,t =|P INI,t -P kNk,t |; Then the distance change rate between finger nodes ΔD ik,t for: ΔD Ik,t =D Ik,t -D Ik,t-1 ; Introduce weighted moving average and calculate the distance change rate ΔD Ik,t To perform smoothing: in: w(τ_t)=e_λ(τ_t); In the formula, w(τ_t) represents the weight function; λ represents the positive attenuation factor; ΔT is the size of the time window considered; τ represents the time when the event in the time series occurs; t represents the current time point; S4: Use convolutional neural networks to classify dynamic gesture features. The model matches the input features with the learned patterns and receives real-time gesture data to output the category of the user's current gesture. S5: Define the mapping rules between gestures and machine control commands. The machine control commands are transmitted to the corresponding application programs to trigger the corresponding functions.

2. The human-computer interaction method based on gesture recognition according to claim 1, characterized in that: In S1, the specific steps of point cloud construction are: S1.1: The 2D image information of the gesture is captured by the RGB camera; S1.2: The depth sensor synchronously records the depth information of the gesture and generates a depth map; S1.3: Calibrate the internal and external parameters of the RGB camera and depth sensor, and align the RGB image and the depth map so that each RGB pixel can find its corresponding depth value; S1.4: Use the intrinsic parameters of the RGB camera and the depth value of each pixel to convert the two-dimensional pixel coordinates into three-dimensional coordinates; S1.5: Add RGB color value to each 3D coordinate point to form point cloud data with color information.

3. The human-computer interaction method based on gesture recognition according to claim 2, characterized in that: In S1.4, the specific steps of transforming three-dimensional coordinates are: Convert the two-dimensional pixel coordinates (u, v) to normalized image plane coordinates (x n ,y n ): In the formula, f x represents the focal length of the camera along the width of the image; f y represents the focal length of the camera along the image height; (c x ,c y ) represents the image center coordinates; (x n ,y n ) represents the coordinates in a coordinate system with the camera optical center as the origin and parallel to the image plane; u represents the horizontal coordinate of the pixel; v represents the vertical coordinate of the pixel; For each normalized coordinate (x n ,y n ), using the depth value Z corresponding to the pixel point, calculate the actual coordinates (X, Y, Z) of the point in three-dimensional space: X=Z·x n ;Y=Z·y n ; And transform the above three-dimensional coordinates from the camera coordinate system to the world coordinate system through rotation and translation transformation: In the formula, Represents the coordinates in the camera coordinate system; Represents the coordinates after conversion to the world coordinate system; R represents the rotation matrix; T represents the translation vector.

4. The human-computer interaction method based on gesture recognition according to claim 2, characterized in that: The key point detection algorithm is based on the hand node model. The specific identification steps for key nodes are: The point cloud is divided into different regions by using the K-means clustering algorithm, at least one of which should contain the hand point cloud; the actual hand point cloud data of the current frame acquired by the depth camera is filtered and denoised by using the mean filter; Predefine the hand node model including the basic structure of wrist, palm, fingers and their key node layout, and align its initial posture with the hand area in the point cloud data; Based on the preliminary alignment, the iterative closest point algorithm is used to iteratively minimize the error from the projection of the hand node model to the point cloud, and the posture of the hand node model is continuously adjusted until the best fit is achieved; Using the preset key node positions in the model, the aligned model is compared with the actual point cloud data, and the coordinates of the key nodes in the real three-dimensional space are located through the nearest neighbor search algorithm.

5. The human-computer interaction method based on gesture recognition according to claim 4, characterized in that: The hand node model is specifically: The hand node model has N key nodes, each of which can be represented by a three-dimensional coordinate (X i ,Y i ,Z i ), where i = 1, 2, ..., N, the hand node model includes the following nodes: Wrist node: P1 = (X wrist ,Y wrist ,Z wrist ); Palm center node: P2 = (X palmC ,Y palmC ,Z palmC ); For each finger, there are three joint nodes and a fingertip node: P 3+j*3+k =(X fingj,k ,Y fingj,k ,Z fingj,k ); where j = 1, 2, 3, 4, 5; k＝1,2,3,4； For the bending angle between any adjacent joints, calculate the bending angle θ between the nth joint and the n+1th joint n , where node P n Represents the position of the nth joint, node P n+1 represents the position of the n+1th joint, then: In the formula, Represents the vector from the nth joint node to the n+1th joint node; represents the vector from the nth joint node to the wrist node; · represents the vector dot product; |·| represents the modulus of the vector.

6. The human-computer interaction method based on gesture recognition according to claim 1, characterized in that: Construct the gesture contour through point cloud data and extract the shape feature vector. The specific steps are as follows: The convex hull algorithm is used to construct the gesture contour through the point cloud data, and the boundary points representing the gesture contour are screened out from the point cloud; Use the directional gradient histogram to represent the shape features of the gesture. For the directional gradient histogram, the steps are: Segment the gesture contour into small cell units; The orientation histogram of pixel gradients is calculated within each cell to capture the local edge orientation distribution; Adjacent cells are combined into blocks, and the directional gradient histogram features of each block are calculated, which are further aggregated into a global descriptor; The gesture shape feature vectors of each frame are concatenated to form a time series feature vector.

7. The human-computer interaction method based on gesture recognition according to claim 6, characterized in that: The convolutional neural network classifies dynamic gesture features. The specific steps are: Normalize the dynamic gesture features and use the sliding window technique to continuously capture the changes in gestures. Each segment is used as an input sample of the convolutional neural network model. Construct a convolutional neural network model and train it using labeled gesture samples and dynamic gesture features; Deploy the trained convolutional neural network model, which receives and processes real-time gesture data and outputs the corresponding gesture category.

8. A human-computer interaction system based on gesture recognition, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the human-computer interaction method based on gesture recognition as described in any one of claims 1-7.

Images