Gesture tracking method and device, computer equipment and readable storage medium

Abstract

The invention relates to the technical field of gesture recognition, and discloses a gesture tracking method and device, computer equipment and a readable storage medium, and the method comprises the steps: respectively obtaining image data collected by a conventional camera and event data collected by an event camera; performing first feature extraction on the image data to obtain conventional camera feature information, and performing second feature extraction on the event data to obtain event camera feature information; inputting the conventional camera feature information and the event camera feature information into a pre-constructed multi-element interactive fusion network to generate fusion data; and tracking the gesture according to the fused data. According to the invention, the conventional camera and the event camera are fused, so that more detailed fusion data can be obtained, the problems of large gesture tracking delay of the conventional camera and poor gesture tracking effect of the event camera are avoided, and high-frame-rate and high-precision tracking of gestures is realized.

Description

Technical Field

[0001] The present invention relates to the field of gesture recognition technology, and in particular to a gesture tracking method, device, computer equipment and readable storage medium. Background Art

[0002] Gesture tracking technology refers to controlling the operation of computers, smartphones and other devices by identifying and tracking the movements of human hands. With the development of technologies such as AI (Artificial Intelligence) and the Internet of Things, gesture tracking technology has become increasingly sophisticated and has a wider range of applications.

[0003] Currently, commonly used gesture tracking methods include: gesture tracking systems based on conventional cameras, but due to the low frame rate of conventional cameras, the performance consumption of processing image data is large, resulting in a large gesture tracking delay; gesture tracking systems based on event cameras, but because event cameras can only capture moving image information, the effect is very poor when the gesture movement amplitude is small. Although there is a combination of conventional cameras and event cameras to recognize gestures in the prior art, it only targets specific types of gestures and has certain limitations, and still cannot achieve a good tracking effect. Summary of the invention

[0004] In view of this, the present invention provides a gesture tracking method, apparatus, computer device and readable storage medium to solve the problem of poor gesture tracking effect in the prior art.

[0005] In a first aspect, the present invention provides a gesture tracking method, the method comprising:

[0006] Acquire image data collected by a conventional camera and event data collected by an event camera respectively;

[0007] Performing a first feature extraction on the image data to obtain conventional camera feature information, and performing a second feature extraction on the event data to obtain event camera feature information;

[0008] Input the conventional camera feature information and the event camera feature information into the pre-built multivariate interactive fusion network to generate fusion data;

[0009] The hand gesture is tracked based on the fused data.

[0010] The present invention provides a gesture tracking method provided by an embodiment, which obtains image data collected by a conventional camera and event data collected by an event camera, performs a first feature extraction on the image data and a second feature extraction on the event data, inputs the obtained conventional camera feature information and event camera feature information into a pre-built multivariate interactive fusion network, generates fusion data, and tracks gestures based on the fusion data. The present invention can obtain more detailed fusion data by fusing a conventional camera with an event camera, avoids the problems of large delay in gesture tracking with a conventional camera and poor gesture tracking effect with an event camera, and achieves high frame rate and high precision tracking of gestures.

[0011] In an optional embodiment, image data collected by a conventional camera and event data collected by an event camera are obtained respectively, including: calibrating the pose of the event camera and the conventional camera to obtain a pose matrix; within a preset sampling time interval, collecting images from the conventional camera at a first sampling frequency to obtain a frame of image data; within a preset sampling time interval, collecting events from the event camera at a second sampling frequency to obtain multiple frames of event data, wherein the second sampling frequency is greater than the first sampling frequency.

[0012] The present invention uses the sampling time of two adjacent frames of image data as the sampling time interval, and performs image acquisition on conventional cameras and event cameras according to different sampling frequencies, so as to obtain the sampling result of one frame of image data for multiple frames of event data, which not only takes into account the static image of the conventional camera, but also takes into account the motion information of the event camera, thereby improving the richness of the data.

[0013] In an optional embodiment, a first feature extraction is performed on the image data to obtain conventional camera feature information, and a second feature extraction is performed on the event data to obtain event camera feature information, including: performing static feature extraction on the image data to obtain an image feature map, and using the image feature map as conventional camera feature information; performing speed feature extraction on multiple frames of event data to obtain multiple frames of event feature maps, and using the multiple frames of event feature maps as multi-frame event camera feature information.

[0014] The present invention extracts features from the acquired image data and event data by taking into account the inherent characteristics of conventional cameras and event cameras, and can take into account both static image features and dynamic speed features, thus facilitating subsequent model data fusion for the extracted features.

[0015] In an optional embodiment, a multivariate interactive fusion network includes: a posture fusion network, a first recurrent neural network and a second recurrent neural network. Conventional camera feature information and event camera feature information are input into a pre-constructed multivariate interactive fusion network to generate fusion data, including: inputting the fusion data generated by the multivariate interactive fusion network in the previous preset sampling time interval and the conventional camera feature information in the current preset sampling time interval into the first recurrent neural network; inputting each frame of event camera feature information and the posture matrix into the posture fusion network in turn; inputting the output result of the first recurrent neural network into the second recurrent neural network, and inputting the output result of each frame of the posture fusion network into the second recurrent neural network in turn, superimposing and fusing the conventional camera feature information and multiple frames of event camera feature information; within the current preset sampling time interval, each frame of event camera feature information is fused to generate the current frame fusion data.

[0016] The present invention fuses conventional camera feature information and event camera feature information through a constructed multivariate interactive fusion network, and based on the sampling frequency of the event camera, generates a frame of fusion data after fusing each frame of event camera feature information. This can solve the problem of low rate of conventional cameras and the problem of poor recognition effect of event cameras for gestures with small motion amplitudes, thereby achieving high frame rate and high-precision gesture tracking.

[0017] In an optional embodiment, a multivariate interactive fusion network is constructed, including: collecting image data and event data with known key point information and degree of freedom information; performing a first feature extraction on the image data to obtain conventional camera feature information, and performing a second feature extraction on the event data to obtain event camera feature information; constructing a multivariate interactive fusion network; inputting the conventional camera feature information within the same preset sampling time interval into a first recurrent neural network, and sequentially inputting the event camera feature information and pose matrix of different frames into a pose fusion network; inputting the output result of the first recurrent neural network into a second recurrent neural network, and sequentially inputting the output result of each frame of the pose fusion network into the second recurrent neural network, and performing a single superposition training on the multivariate interactive fusion network; inputting the training result of the second recurrent neural network at the current preset sampling time interval into the first recurrent neural network at the next preset sampling time interval, and performing a cyclic training on the multivariate interactive fusion network to obtain a final multivariate interactive fusion network.

[0018] The present invention uses the sampling time of two adjacent frames of image data as the sampling time interval, performs feature extraction and data fusion on one frame of image data and multiple frames of event data within the same sampling time interval, and can combine the inherent characteristics of conventional cameras and event cameras to perform single superposition training on the multivariate interactive fusion network. Then, according to the training results of different sampling time intervals, cyclic training is performed to predict fused data with time-continuous characteristics, and then continuous tracking of gestures can be achieved based on the fused data.

[0019] In an optional embodiment, tracking gestures based on fusion data includes: obtaining a hand skeleton framework; inputting the hand skeleton framework and each frame of fusion data in each preset sampling time interval into a pre-built decoding network in sequence to obtain posture data of different frames in different preset sampling time intervals; determining the hand state based on the current frame posture data, and tracking gestures based on the hand states in different frames.

[0020] The present invention can obtain hand posture data by decoding the fusion data based on the hand skeleton framework, so that the hand gesture can be continuously and efficiently tracked according to the posture data.

[0021] In an optional embodiment, the hand skeleton framework and each frame of fusion data in each preset sampling time interval are sequentially input into a pre-constructed decoding network to obtain posture data of different frames in different preset sampling time intervals, including: decoding each frame of fusion data in each preset sampling time interval according to the hand skeleton framework to obtain posture data of the corresponding frame, the posture data including: key point information and degree of freedom information, wherein the preset key point information is 2D information or 3D information of a preset position of the hand, and the degree of freedom information includes: wrist position information, wrist rotation information and finger curvature information.

[0022] By obtaining the key point information and degree of freedom information of the hand, the present invention can determine the position information of each key point of each finger and the rotation or bending information of the wrist and fingers, and further determine the current state of the hand.

[0023] In a second aspect, the present invention provides a gesture tracking device, the device comprising:

[0024] A data acquisition module, used to respectively acquire image data acquired by a conventional camera and event data acquired by an event camera;

[0025] A feature extraction module is used to perform a first feature extraction on the image data to obtain conventional camera feature information, and to perform a second feature extraction on the event data to obtain event camera feature information;

[0026] A data fusion module is used to input the feature information of conventional cameras and event cameras into a pre-built multivariate interactive fusion network to generate fused data;

[0027] The gesture tracking module is used to track the gesture based on the fused data.

[0028] The gesture tracking device provided by the embodiment of the present invention obtains image data collected by a conventional camera and event data collected by an event camera, performs a first feature extraction on the image data and a second feature extraction on the event data, inputs the obtained conventional camera feature information and event camera feature information into a pre-built multivariate interactive fusion network, generates fusion data, and tracks gestures based on the fusion data. The present invention can obtain more detailed fusion data by fusing a conventional camera with an event camera, avoids the problems of large delay in gesture tracking with a conventional camera and poor gesture tracking effect with an event camera, and achieves high frame rate and high precision tracking of gestures.

[0029] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the gesture tracking method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0030] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the gesture tracking method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 is a flowchart of a gesture tracking method according to an embodiment of the present invention;

[0033] Figure 2 is a flow chart of another gesture tracking method according to an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of camera fixed connection according to another gesture tracking method according to an embodiment of the present invention;

[0035] Figure 4is a schematic diagram of a data timeline of another gesture tracking method according to an embodiment of the present invention;

[0036] Figure 5 is a schematic diagram of a specific flow chart of another gesture tracking method according to an embodiment of the present invention;

[0037] Figure 6 is a schematic diagram of a model structure of another gesture tracking method according to an embodiment of the present invention;

[0038] Figure 7 is a schematic diagram of a posture fusion network of another gesture tracking method according to an embodiment of the present invention;

[0039] Figure 8 is a flowchart of another gesture tracking method according to an embodiment of the present invention;

[0040] Fig. 9 is a schematic diagram of a decoding network of another gesture tracking method according to an embodiment of the present invention;

[0041] Fig.10 is a schematic diagram of key position information of another gesture tracking method according to an embodiment of the present invention;

[0042] Fig.11 is a schematic diagram of degree of freedom information of another gesture tracking method according to an embodiment of the present invention;

[0043] Fig.12 is a structural block diagram of a gesture tracking device according to an embodiment of the present invention;

[0044] Fig.13 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] The embodiment of the present invention is applicable to the scene of tracking hand movements, and provides a gesture tracking method, which achieves the effect of high frame rate and high precision tracking of gestures by fusing conventional cameras and event cameras. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0047] In this embodiment, a gesture tracking method is provided, which can be used in the above-mentioned computer. Figure 1 is a flowchart of a gesture tracking method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0048] Step S101, respectively acquiring image data collected by a conventional camera and event data collected by an event camera.

[0049] Specifically, in the embodiment of the present invention, the conventional camera has clear imaging, and an RGB color camera, a monochrome grayscale camera or an RGBD depth camera may be used, but is not limited thereto. The event camera is a new type of biologically inspired visual sensor. Compared with conventional cameras that capture images at a fixed frame rate, event cameras have the characteristics of low latency, high dynamic range, low power consumption, and high temporal resolution. In the embodiment of the present invention, the image data captured by the conventional camera is used as the key frame, and the event data captured by the event camera is used as the ordinary frame. The two cameras are asynchronous when collecting data. The conventional camera has a low sampling frequency, and the event camera has a high sampling frequency. Usually, one frame in the conventional camera corresponds to multiple event frames in the event camera.

[0050] Step S102, performing a first feature extraction on the image data to obtain conventional camera feature information, and performing a second feature extraction on the event data to obtain event camera feature information.

[0051] Specifically, in the embodiment of the present invention, the acquired image data and event data are raw data, including irrelevant data. Therefore, the embodiment of the present invention extracts features from the image data and event data respectively according to the characteristics of the conventional camera and the event camera, selects effective information useful for gesture tracking, obtains conventional camera feature information and event camera feature information respectively, and reduces the complexity of the data.

[0052] Step S103: input the regular camera feature information and the event camera feature information into a pre-built multivariate interactive fusion network to generate fusion data.

[0053] Specifically, in the embodiment of the present invention, a multivariate interactive fusion network with a time series network structure is constructed in advance based on a deep learning network structure, and then the conventional camera feature information corresponding to the low-frequency image data and the event camera feature information corresponding to the high-frequency event data are input into the multivariate interactive fusion network. The multivariate interactive fusion network uses the conventional camera feature information as the basic input and the event camera feature information as the incremental input, performs fusion processing on the image data and the event data, and generates fusion data with time series features.

[0054] Step S104: Track the gesture according to the fused data.

[0055] Specifically, in the embodiment of the present invention, tracking gestures is not limited to identifying specific gestures, but can determine the state of the hand at each moment based on fusion data with time series characteristics, thereby achieving continuous tracking of hand movements.

[0056] The present invention provides a gesture tracking method provided by an embodiment, which obtains image data collected by a conventional camera and event data collected by an event camera, performs a first feature extraction on the image data and a second feature extraction on the event data, inputs the obtained conventional camera feature information and event camera feature information into a pre-constructed multivariate interactive fusion network, generates fusion data, and tracks gestures based on the fusion data. The present invention can obtain more detailed fusion data by fusing a conventional camera with an event camera, avoids the problems of large delay in gesture tracking with a conventional camera and poor gesture tracking effect with an event camera, achieves high frame rate and high precision tracking of gestures, and can achieve low power consumption and low CPU occupancy.

[0057] In this embodiment, a gesture tracking method is provided, which can be used in the above-mentioned computer. Figure 2 is a flowchart of a gesture tracking method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0058] Step S201, respectively acquiring image data collected by a conventional camera and event data collected by an event camera.

[0059] Specifically, the above step S201 includes:

[0060] Step S2011, calibrate the pose of the event camera and the conventional camera to obtain a pose matrix.

[0061] Step S2012: within a preset sampling time interval, the conventional camera is used to collect images according to a first sampling frequency to obtain a frame of image data.

[0062] Step S2013: within a preset sampling time interval, the event camera is used to collect events at a second sampling frequency to obtain multiple frames of event data, wherein the second sampling frequency is greater than the first sampling frequency.

[0063] Specifically, in the embodiment of the present invention, Figure 3As shown, the conventional camera and the event camera are fixed by a connecting device, and then the conventional camera and the event camera are jointly calibrated to obtain the pose matrix of the event camera to the conventional camera. The conventional camera acquires images at a first sampling frequency of 30 frames / second, and the event camera acquires events at a second sampling frequency of 120 frames / second, but not limited to this. It can be seen that the second sampling frequency of the event camera is greater than the first sampling frequency of the conventional camera, so within the preset sampling time interval of the sampling time between the image data of two adjacent frames, it contains one frame of image data and multiple frames of time data.

[0064] Step S202: performing a first feature extraction on the image data to obtain conventional camera feature information, and performing a second feature extraction on the event data to obtain event camera feature information.

[0065] Specifically, the above step S202 includes:

[0066] Step S2021, perform static feature extraction on the image data to obtain an image feature map, and use the image feature map as regular camera feature information.

[0067] Step S2022, extracting speed features from the multi-frame event data respectively to obtain a multi-frame event feature map, and using the multi-frame event feature map as multi-frame event camera feature information.

[0068] Specifically, in the embodiment of the present invention, static feature extraction is performed on the image data according to the inherent characteristics of the conventional camera, and speed feature extraction is performed on the event data according to the inherent characteristics of the event camera, so that one frame of conventional camera feature information and multiple frames of event camera feature information are included in a preset sampling time interval. The feature extraction of the embodiment of the present invention can use a CNN convolutional neural network or other feature extraction networks to extract key feature information of the image from the input image information and generate an image feature map.

[0069] Step S203: input the regular camera feature information and the event camera feature information into a pre-built multivariate interactive fusion network to generate fusion data.

[0070] Specifically, the above step S203 includes:

[0071] Step S2031, inputting the fusion data generated by the multivariate interactive fusion network in the previous preset sampling time interval and the conventional camera feature information in the current preset sampling time interval into the first recurrent neural network.

[0072] Step S2032, input each frame event camera feature information and pose matrix into the pose fusion network in sequence.

[0073] Step S2033, input the output result of the first recurrent neural network into the second recurrent neural network, and input the output result of each frame pose fusion network into the second recurrent neural network in sequence, and superimpose and fuse the conventional camera feature information and the multi-frame event camera feature information.

[0074] Step S2034: within the current preset sampling time interval, each frame of event camera feature information is fused to generate current frame fusion data.

[0075] Specifically, in the embodiment of the present invention, Figure 4 and Figure 5 As shown in , the conventional camera feature information is used as the key frame, and the conventional camera key frame and event camera data frame are input into the pre-built multi-element interactive fusion network in the time order represented by the time axis. At the same time, the previous frame output of the multi-element interactive fusion network is also used as the input of the current frame. Figure 6 As shown, the multi-element interactive fusion network includes: a posture fusion network, a first recurrent neural network (RNN) and a second recurrent neural network, wherein the RNN can be a CNN module with a recurrent structure, or a network structure with memory function such as LSTM, GRU, Transformer, etc., which is not limited here.

[0076] In some optional implementations, taking a preset sampling time interval containing one frame of conventional camera feature information and three frames of event camera feature information as an example, the embodiment of the present invention inputs the fusion data generated by the multivariate interactive fusion network in the previous preset sampling time interval and one frame of conventional camera feature information in the current preset sampling time interval into the first recurrent neural network, and at the same time, sequentially inputs each frame of data and the pose matrix of the three frames of event camera feature information into the pose fusion network. Then, the output result of the first recurrent neural network is input into the second recurrent neural network, and the output result of each frame of the pose fusion network is sequentially input into the second recurrent neural network, and the conventional camera feature information and the multi-frame event camera feature information are superimposed and fused. This is equivalent to first fusing one frame of conventional camera feature information that has passed through the first recurrent neural network with the first frame of event camera feature information that has passed through the pose fusion network, and then fusing the fusion result with the second frame of event camera feature information that has passed through the pose fusion network, and so on, until the superposition and fusion of one frame of conventional camera feature information and three frames of event camera feature information in the current preset sampling time interval is completed. In the superposition fusion process of the embodiment of the present invention, within each preset sampling time interval, the feature information of each frame of the event camera is fused to generate the current frame fusion data, that is, three frames of fusion data are generated within a preset sampling time interval, and the frame rate of the fusion data is the same as the frame rate of the event camera.

[0077] In some optional embodiments, the process of constructing a multivariate interactive fusion network in an embodiment of the present invention includes: (1) collecting image data and event data with known key point information and degree of freedom (Dof) information; (2) performing a first feature extraction on the image data to obtain conventional camera feature information, and performing a second feature extraction on the event data to obtain event camera feature information; (3) constructing a multivariate interactive fusion network; (4) inputting the conventional camera feature information within the same preset sampling time interval into the first recurrent neural network, and sequentially inputting the event camera feature information and pose matrix of different frames into the pose fusion network; (5) inputting the output result of the first recurrent neural network into the second recurrent neural network, and sequentially inputting the output result of each frame of the pose fusion network into the second recurrent neural network, and performing a single superposition training on the multivariate interactive fusion network; (6) inputting the training result of the second recurrent neural network at the current preset sampling time interval into the first recurrent neural network at the next preset sampling time interval, and performing a cyclic training on the multivariate interactive fusion network to obtain the final multivariate interactive fusion network. The training process of the multivariate interactive fusion network in the embodiment of the present invention is consistent with the prediction process. Among them, if Figure 7 As shown, the pose fusion network includes matrix cross product and MLP layer, which can convert the event camera feature information into feature information corresponding to the conventional camera pose. The construction of the pose fusion network is a conventional technical means in this field and will not be repeated here.

[0078] Step S204: Track the gesture based on the fused data. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0079] The present invention provides a gesture tracking method provided by an embodiment, which obtains image data collected by a conventional camera and event data collected by an event camera, performs a first feature extraction on the image data and a second feature extraction on the event data, inputs the obtained conventional camera feature information and event camera feature information into a pre-built multivariate interactive fusion network, generates fusion data, and tracks gestures based on the fusion data. The present invention can obtain more detailed fusion data by fusing a conventional camera with an event camera, avoids the problems of large delay in gesture tracking with a conventional camera and poor gesture tracking effect with an event camera, and achieves high frame rate and high precision tracking of gestures.

[0080] In this embodiment, a gesture tracking method is provided, which can be used in the above-mentioned computer. Figure 8 is a flowchart of a gesture tracking method according to an embodiment of the present invention. Figure 8 As shown, the process includes the following steps:

[0081] Step S801, respectively obtain image data collected by a conventional camera and event data collected by an event camera. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0082] Step S802: Perform a first feature extraction on the image data to obtain conventional camera feature information, and perform a second feature extraction on the event data to obtain event camera feature information. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0083] Step S803: Input the regular camera feature information and event camera feature information into the pre-built multivariate interactive fusion network to generate fusion data. Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.

[0084] Step S804: Track the gesture according to the fused data.

[0085] Specifically, the above step 804 includes:

[0086] Step S8041, obtain the hand skeleton framework.

[0087] Step S8042, input the hand skeleton framework and each frame of fusion data in each preset sampling time interval into a pre-built decoding network in sequence, and obtain the posture data of different frames in different preset sampling time intervals.

[0088] Step S8043, determining the hand state according to the current frame posture data, and tracking the gesture according to the hand states in different frames.

[0089] Specifically, in the embodiment of the present invention, the following is pre-constructed: Fig. 9 The decoding network shown includes multiple attention layers and MLP layers, which are specifically constructed as conventional technical means in the field and will not be repeated here. Among them, the attention layer (AttentionLayer) is a key component in the deep learning model, which allows the model to dynamically focus on different parts of the input when processing input data, rather than fixedly processing the entire input. The attention layer allows the model to assign different weights to different positions in the input sequence, and these weights indicate the degree of attention the model pays to the input. In this way, the model can selectively focus on certain parts of the input, thereby improving model performance. Multilayer Perceptron (MLP) is an artificial neural network with a forward structure, comprising an input layer, an output layer and multiple hidden layers. The preset key point information is 2D information or 3D information of the preset position of the hand.

[0090] In some optional implementations, the decoding network of the embodiment of the present invention decodes each frame of fusion data within each preset sampling time interval according to the hand skeleton framework to obtain the posture data of the corresponding frame, and the posture data includes: key point information and degree of freedom information. Fig.10 As shown, the embodiment of the present invention pre-sets 0-23 key positions, which is only an example and is not limited to this. The degree of freedom information includes: wrist position information, wrist rotation information and finger bending information, such as Fig.11 As shown, the embodiment of the present invention is set to 26 degrees of freedom information, where a20 and a21 represent 3D coordinate information and 3D rotation information respectively, and a0-a19 represent the bending information of different joints of each finger respectively, which can be represented by a vector as [1.2, 0.5, 3.0, 0.8, 1.5, 2.7, 20.0, 10.0, 30.0, 15.0, 0.0, 90.0, 75.0, 80.0, 30.0, 60.0, 0.0, 10.0, 5.0, 8.0, 25.0, 45.0, 55.0, 30.0, 20. 0,15.0,5.0], the first 6 dimensions ([1.2,0.5,3.0,0.8,1.5,2.7]) describe the wrist position and rotation information, and the subsequent 20 dimensions ([20.0,10.0,30.0,15.0,0.0,90.0,75.0,80.0,30.0,60.0,0.0,10.0,5.0,8.0,25.0,45.0,55.0,30.0,20.0,15.0,5.0]) represent the bending information of the five fingers, which is only used as an example and is not limited to this. The 26-degree-of-freedom vector can describe the posture and shape of the hand, and thus be applied to gesture recognition. . It can be seen that after obtaining the position information of the key positions of the hand and the rotation and bending information of the wrist and fingers, the hand state can be determined, for example, the thumb and index finger of the "OK" gesture are in a bent state, and the other three fingers are in a straight state. The embodiment of the present invention can obtain time-continuous posture data based on the fusion data of continuous sampling intervals and continuous frames, and then determine the time-continuous hand state based on the time-continuous posture data to achieve gesture tracking.

[0091] The present invention provides a gesture tracking method provided by an embodiment, which obtains image data collected by a conventional camera and event data collected by an event camera, performs a first feature extraction on the image data and a second feature extraction on the event data, inputs the obtained conventional camera feature information and event camera feature information into a pre-built multivariate interactive fusion network, generates fusion data, and tracks gestures based on the fusion data. The present invention can obtain more detailed fusion data by fusing a conventional camera with an event camera, avoids the problems of large delay in gesture tracking with a conventional camera and poor gesture tracking effect with an event camera, and achieves high frame rate and high precision tracking of gestures.

[0092] In this embodiment, a gesture tracking device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0093] This embodiment provides a gesture tracking device, such as Fig.12 As shown, including:

[0094] The data acquisition module 1201 is used to respectively acquire image data acquired by a conventional camera and event data acquired by an event camera;

[0095] The feature extraction module 1202 is used to perform a first feature extraction on the image data to obtain conventional camera feature information, and perform a second feature extraction on the event data to obtain event camera feature information;

[0096] The data fusion module 1203 is used to input the conventional camera feature information and the event camera feature information into a pre-built multivariate interactive fusion network to generate fused data;

[0097] The gesture tracking module 1204 is used to track the gesture according to the fused data.

[0098] In some optional implementations, the data acquisition module 1201 includes:

[0099] The pose calibration unit is used to calibrate the pose of the event camera and the conventional camera to obtain a pose matrix.

[0100] The first acquisition unit is used to acquire images from a conventional camera at a first sampling frequency within a preset sampling time interval to obtain a frame of image data.

[0101] The second acquisition unit is used to acquire events from the event camera at a second sampling frequency within a preset sampling time interval to obtain multiple frames of event data, wherein the second sampling frequency is greater than the first sampling frequency.

[0102] In some optional implementations, the feature extraction module 1202 includes:

[0103] The first feature extraction unit is used to perform static feature extraction on the image data to obtain an image feature map, and use the image feature map as conventional camera feature information.

[0104] The second feature extraction unit is used to extract speed features from the multi-frame event data respectively, obtain a multi-frame event feature map, and use the multi-frame event feature map as the multi-frame event camera feature information.

[0105] In some optional embodiments, the multivariate interactive fusion network of the data fusion module 1203 includes: a posture fusion network, a first recurrent neural network, and a second recurrent neural network. The conventional camera feature information and the event camera feature information are input into a pre-constructed multivariate interactive fusion network to generate fusion data, including: inputting the fusion data generated by the multivariate interactive fusion network in the previous preset sampling time interval and the conventional camera feature information in the current preset sampling time interval into the first recurrent neural network; inputting each frame of event camera feature information and the posture matrix into the posture fusion network in turn; inputting the output result of the first recurrent neural network into the second recurrent neural network, and inputting the output result of each frame of the posture fusion network into the second recurrent neural network in turn, superimposing and fusing the conventional camera feature information and multiple frames of event camera feature information; within the current preset sampling time interval, each frame of event camera feature information is fused to generate the current frame fusion data.

[0106] In some optional implementations, the data fusion module 1203 includes:

[0107] A model building unit is used to collect image data and event data with known key point information and degree of freedom information; perform a first feature extraction on the image data to obtain conventional camera feature information, and perform a second feature extraction on the event data to obtain event camera feature information; construct a multivariate interactive fusion network; input the conventional camera feature information within the same preset sampling time interval into the first recurrent neural network, and input the event camera feature information and posture matrix of different frames into the posture fusion network in turn; input the output result of the first recurrent neural network into the second recurrent neural network, and input the output result of each frame of the posture fusion network into the second recurrent neural network in turn, and perform a single superposition training on the multivariate interactive fusion network; input the training result of the second recurrent neural network at the current preset sampling time interval into the first recurrent neural network at the next preset sampling time interval, perform cyclic training on the multivariate interactive fusion network, and obtain the final multivariate interactive fusion network.

[0108] In some optional implementations, the gesture tracking module 1204 includes:

[0109] The hand skeleton frame acquisition unit is used to acquire the hand skeleton frame.

[0110] The posture data generating unit is used to sequentially input the hand skeleton framework and each frame of fusion data in each preset sampling time interval into a pre-built decoding network to obtain the posture data of different frames in different preset sampling time intervals.

[0111] The tracking unit is used to determine the hand state according to the current frame posture data and track the gesture according to the hand state in different frames.

[0112] In some optional embodiments, the posture data generation unit includes: decoding each frame of fusion data within each preset sampling time interval according to the hand skeleton framework to obtain posture data of the corresponding frame, the posture data including: key point information and degree of freedom information, wherein the preset key point information is 2D information or 3D information of the preset position of the hand, and the degree of freedom information includes: wrist position information, wrist rotation information and finger curvature information.

[0113] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0114] The gesture tracking device in this embodiment is presented in the form of a functional unit, where the unit refers to an FPGA (Field Programmable Gate Array) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0115] The embodiment of the present invention also provides a computer device having the above Fig.12 The gesture tracking device shown.

[0116] See also Fig.13 , Fig.13 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.13 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.13 A processor 10 is taken as an example.

[0117] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0118] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0119] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0121] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0122] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0123] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A gesture tracking method, characterized in that: The method comprises: Acquire image data collected by a conventional camera and event data collected by an event camera respectively; Performing a first feature extraction on the image data to obtain conventional camera feature information, and performing a second feature extraction on the event data to obtain event camera feature information; Inputting the conventional camera feature information and the event camera feature information into a pre-built multivariate interactive fusion network to generate fusion data; The gesture is tracked based on the fused data.

2. The method according to claim 1, characterized in that The step of respectively acquiring the image data collected by the conventional camera and the event data collected by the event camera comprises: Calibrate the pose of the event camera and the conventional camera to obtain a pose matrix; Within a preset sampling time interval, performing image acquisition on the conventional camera according to a first sampling frequency to obtain a frame of image data; Within a preset sampling time interval, the event camera is used to collect events according to a second sampling frequency to obtain a plurality of frames of event data, wherein the second sampling frequency is greater than the first sampling frequency.

3. The method according to claim 2, characterized in that The first feature extraction is performed on the image data to obtain conventional camera feature information, and the second feature extraction is performed on the event data to obtain event camera feature information, including: Performing static feature extraction on the image data to obtain an image feature map, and using the image feature map as conventional camera feature information; Speed ​​features are extracted from the multiple frames of event data respectively to obtain a multi-frame event feature map, and the multi-frame event feature map is used as multi-frame event camera feature information.

4. The method according to claim 3, characterized in that The multivariate interactive fusion network includes: a posture fusion network, a first recurrent neural network, and a second recurrent neural network. The conventional camera feature information and the event camera feature information are input into the pre-constructed multivariate interactive fusion network to generate fusion data, including: Inputting the fusion data generated by the multivariate interactive fusion network in the last preset sampling time interval and the conventional camera feature information in the current preset sampling time interval into the first recurrent neural network; Inputting the event camera feature information and the pose matrix of each frame into the pose fusion network in sequence; Inputting the output result of the first recurrent neural network into the second recurrent neural network, and sequentially inputting the output result of the pose fusion network of each frame into the second recurrent neural network, superimposing and fusing the conventional camera feature information and the multi-frame event camera feature information; Within the current preset sampling time interval, each frame of the event camera feature information is fused to generate current frame fusion data.

5. The method according to claim 4, characterized in that Constructing the multivariate interactive fusion network includes: Collect the image data and the event data with known key point information and degree of freedom information; Performing a first feature extraction on the image data to obtain conventional camera feature information, and performing a second feature extraction on the event data to obtain event camera feature information; Build a multi-dimensional interactive fusion network; Inputting the conventional camera feature information within the same preset sampling time interval into the first recurrent neural network, and sequentially inputting the event camera feature information and the pose matrix of different frames into the pose fusion network; Input the output result of the first recurrent neural network into the second recurrent neural network, and input the output result of the posture fusion network of each frame into the second recurrent neural network in sequence, and perform single superposition training on the multivariate interactive fusion network; The training result of the second recurrent neural network at the current preset sampling time interval is input into the first recurrent neural network at the next preset sampling time interval, and the multivariate interactive fusion network is cyclically trained to obtain a final multivariate interactive fusion network.

6. The method according to claim 4, characterized in that Tracking the gesture according to the fusion data includes: Get the hand skeleton frame; Inputting the hand skeleton framework and each frame of the fusion data in each preset sampling time interval into a pre-constructed decoding network in sequence to obtain posture data of different frames in different preset sampling time intervals; The hand state is determined according to the posture data of the current frame, and the gesture is tracked according to the hand states of different frames.

7. The method according to claim 6, characterized in that The step of sequentially inputting the hand skeleton framework and each frame of the fusion data within each preset sampling time interval into a pre-built decoding network to obtain posture data of different frames within different preset sampling time intervals includes: The fusion data of each frame within each preset sampling time interval is decoded according to the hand skeleton framework to obtain posture data of the corresponding frame, wherein the posture data includes: key point information and degree of freedom information, wherein the preset key point information is 2D information or 3D information of a preset position of the hand, and the degree of freedom information includes: wrist position information, wrist rotation information and finger curvature information.

8. A gesture tracking device, characterized in that: The device comprises: A data acquisition module, used to respectively acquire image data acquired by a conventional camera and event data acquired by an event camera; A feature extraction module, configured to perform a first feature extraction on the image data to obtain conventional camera feature information, and perform a second feature extraction on the event data to obtain event camera feature information; A data fusion module, used for inputting the conventional camera feature information and the event camera feature information into a pre-built multivariate interactive fusion network to generate fused data; The gesture tracking module is used to track the gesture according to the fused data.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the gesture tracking method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the gesture tracking method according to any one of claims 1 to 7.

Images