Binocular camera high-frame-rate gesture tracking method and system

Abstract

The invention provides a binocular camera high-frame-rate gesture tracking method and system, and relates to the technical field of gesture tracking. The gesture tracking method comprises the following steps: acquiring pose data of a first camera and a second camera based on a preset binocular camera; alternately sampling images frame by frame through the first camera and the second camera to obtain a binocular camera image frame sequence to be analyzed; taking the binocular camera image frame sequence to be analyzed and the binocular camera pose data as input of a pre-trained binocular asynchronous neural network, and outputting 3D key point coordinates of each frame of gesture as a gesture tracking result; according to the gesture tracking method, in the processing of two adjacent frames, the same processing is carried out on a single camera image of a certain frame, so that only one-time operation is needed, the result of the first-time reuse is carried out for the second time, and the calculation complexity, the CPU occupancy rate and the power consumption are greatly reduced; the processing time is shortened, the high-frame-rate reasoning speed can be realized, and a quick and high-precision gesture tracking algorithm is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture tracking, and in particular to a high frame rate gesture tracking method and system for a binocular camera. Background Art

[0002] In recent years, the development of gesture tracking technology has benefited from the progress in computer vision, image processing, pattern recognition and machine learning. With the improvement of computer hardware performance and the continuous innovation of sensor technology, gesture tracking technology has made significant progress.

[0003] Traditional gesture tracking technology is mainly based on capturing gestures with cameras, and then using image processing algorithms to analyze the shape, position and movement of gestures. With the rise of deep learning technology, the application of neural networks in gesture tracking has gradually increased, enabling the system to track gestures more accurately.

[0004] Existing gesture tracking technology uses two cameras to shoot at 30 frames per second, first uses a convolutional neural network (CNN) to generate single-purpose 2D key points, then calculates the parallax between the two cameras to obtain depth information, thereby deriving 3D gesture key points, and then generating 3D gesture key points. This method can help the system capture the three-dimensional position of the gesture more accurately.

[0005] However, this method has high computational complexity, consumes a lot of performance and time to process image data, resulting in large tracking delays. The high computational complexity causes the system to only run at a frequency of 30 frames per second, making the gesture tracking delay more than 30 milliseconds. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the deficiencies in the prior art, the present invention provides a binocular camera high frame rate gesture tracking method and system, which solves the problems of high computational complexity and high latency in gesture tracking methods.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] A high frame rate gesture tracking method for a binocular camera, the gesture tracking method comprising the following steps:

[0011] Based on a preset binocular camera, obtaining the position and posture data of the first camera and the second camera;

[0012] Acquire a binocular camera image frame sequence to be analyzed by alternately sampling images frame by frame with the first camera and the second camera;

[0013] The binocular camera image frame sequence and binocular camera pose data to be analyzed are used as the input of the pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result.

[0014] Preferably, the training of the binocular asynchronous neural network comprises the following steps:

[0015] The first camera and the second camera simultaneously sample images at the same frequency, and based on the image of each frame and the binocular camera posture data, the coordinates of the 3D key points of the gesture in each frame are obtained;

[0016] The first camera image and the second camera image are discarded frame by frame alternately from the images sampled simultaneously by the first camera and the second camera at the same frequency, to obtain multiple groups of binocular camera image frame sequences;

[0017] Each set of binocular camera image frame sequence and binocular camera pose data is used as a training sample, and the corresponding 3D key point coordinates of each frame of gesture are used as supervision signals to build a training database;

[0018] Network training is performed based on the training database to obtain a binocular asynchronous neural network.

[0019] Preferably, the binocular asynchronous neural network includes: an image encoding network, a pose projection network, a time alignment network, a feature fusion layer, a timing network and a decoding network;

[0020] The image encoding network adopts a convolutional neural network;

[0021] And / or the time alignment network is a multi-layer perceptron structure;

[0022] And / or the feature fusion layer is a convolutional layer;

[0023] And / or the timing network is an RNN recurrent neural network, and the RNN recurrent neural network includes: one or more of a CNN module with a recurrent structure, an LSTM network, a GRU network, and a Transformer network;

[0024] And / or the decoding network is a superposition of an attention layer and a multi-layer perceptron layer.

[0025] Preferably, the binocular camera image frame sequence and binocular camera posture data to be analyzed are used as inputs of a pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result, including the following steps:

[0026] The first camera image or the second camera image of the previous frame is converted into a feature map through an image encoding network, and the feature map and the corresponding camera pose data are converted into a 3D pose projection through a pose projection network. The 3D pose projection is aligned to the pose processed by the previous frame image to the input feature fusion layer of the current frame through a time alignment network;

[0027] The second camera image or the first camera image of the current frame is processed by the image encoding network and the corresponding camera pose data is input into the pose projection network, and then input into the feature fusion layer after being processed by the pose projection network;

[0028] The feature fusion layer fuses the two input 3D posture projections into a unified 3D posture projection input timing network;

[0029] The temporal network stores the feature information after inference of each frame and transmits it to the next frame inference network as input to maintain the continuity of the previous and next frame data;

[0030] The decoding network decodes the feature data and outputs the coordinates of the 3D key points of each frame of gesture.

[0031] Preferably, the binocular camera comprises: a first camera, a second camera and a connecting device;

[0032] The first camera is fixedly connected to the second camera via a connecting device;

[0033] The first camera and the second camera are RGB cameras or grayscale cameras of the same model.

[0034] A binocular camera high frame rate gesture tracking system, the gesture tracking system comprising: a posture acquisition module, an image frame sequence acquisition module and a gesture tracking module;

[0035] The posture acquisition module is used to acquire the posture data of the first camera and the second camera;

[0036] The image frame sequence acquisition module is used to acquire a binocular camera image frame sequence to be analyzed by alternately sampling images frame by frame through the first camera and the second camera;

[0037] The gesture tracking module is used to use the binocular camera image frame sequence to be analyzed and the binocular camera posture data as the input of the pre-trained binocular asynchronous neural network, and output the 3D key point coordinates of each frame of gesture as the gesture tracking result.

[0038] Preferably, the training of the binocular asynchronous neural network comprises the following steps:

[0039] The first camera and the second camera simultaneously sample images at the same frequency, and based on the image of each frame and the binocular camera posture data, the coordinates of the 3D key points of the gesture in each frame are obtained;

[0040] The first camera image and the second camera image are discarded frame by frame alternately from the images sampled simultaneously by the first camera and the second camera at the same frequency, to obtain multiple groups of binocular camera image frame sequences;

[0041] Each set of binocular camera image frame sequence and binocular camera pose data is used as a training sample, and the corresponding 3D key point coordinates of each frame of gesture are used as supervision signals to build a training database;

[0042] Network training is performed based on the training database to obtain a binocular asynchronous neural network.

[0043] Preferably, the binocular asynchronous neural network includes: an image encoding network, a pose projection network, a time alignment network, a feature fusion layer, a timing network and a decoding network;

[0044] The image encoding network adopts a convolutional neural network;

[0045] And / or the time alignment network is a multi-layer perceptron structure;

[0046] And / or the feature fusion layer is a convolutional layer;

[0047] And / or the timing network is an RNN recurrent neural network, and the RNN recurrent neural network includes: one or more of a CNN module with a recurrent structure, an LSTM network, a GRU network, and a Transformer network;

[0048] And / or the decoding network is a superposition of an attention layer and a multi-layer perceptron layer.

[0049] Preferably, the binocular camera image frame sequence and binocular camera posture data to be analyzed are used as inputs of a pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result, including the following steps:

[0050] The first camera image or the second camera image of the previous frame is converted into a feature map through an image encoding network, and the feature map and the corresponding camera pose data are converted into a 3D pose projection through a pose projection network. The 3D pose projection is aligned to the pose processed by the previous frame image to the input feature fusion layer of the current frame through a time alignment network;

[0051] The second camera image or the first camera image of the current frame is processed by the image encoding network and the corresponding camera pose data is input into the pose projection network, and then input into the feature fusion layer after being processed by the pose projection network;

[0052] The feature fusion layer fuses the two input 3D posture projections into a unified 3D posture projection input timing network;

[0053] The temporal network stores the feature information after inference of each frame and transmits it to the next frame inference network as input to maintain the continuity of the previous and next frame data;

[0054] The decoding network decodes the feature data and outputs the coordinates of the 3D key points of each frame of gesture.

[0055] Preferably, the binocular camera comprises: a first camera, a second camera and a connecting device;

[0056] The first camera is fixedly connected to the second camera via a connecting device;

[0057] The first camera and the second camera are RGB cameras or grayscale cameras of the same model.

[0058] (III) Beneficial effects

[0059] The present invention provides a binocular camera high frame rate gesture tracking method and system. Compared with the prior art, it has the following beneficial effects:

[0060] In the present invention, the gesture tracking method includes the following steps: based on a preset binocular camera, obtaining the posture data of a first camera and a second camera; obtaining a binocular camera image frame sequence to be analyzed by alternately sampling images frame by frame through the first camera and the second camera; using the binocular camera image frame sequence to be analyzed and the binocular camera posture data as inputs of a pre-trained binocular asynchronous neural network, and outputting the 3D key point coordinates of each frame of gesture as a gesture tracking result; in the processing of two adjacent frames, the gesture tracking method will perform the same processing on a single camera image of a certain frame, so it only needs to be run once, and the result of the first pass is reused for the second time, which greatly reduces the calculation complexity, CPU occupancy and power consumption; shortens the processing time, can achieve a high frame rate inference speed, and realizes a fast and high-precision gesture tracking algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 is a flowchart of the gesture tracking method in Embodiment 1 and Embodiment 2 of the present invention;

[0063] Figure 2 It is a structural diagram of the binocular asynchronous neural network in Embodiment 1 and Embodiment 2 of the present invention;

[0064] Figure 3Schematic diagram of the structure of the binocular camera in Example 1 and Example 2 of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The embodiments of the present application solve the problems of high computational complexity and high latency in gesture tracking methods by providing a binocular camera high frame rate gesture tracking method and system.

[0067] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0068] In an embodiment of the present invention, the gesture tracking method includes the following steps: based on a preset binocular camera, obtaining the posture data of a first camera and a second camera; alternately sampling images frame by frame by the first camera and the second camera to obtain a binocular camera image frame sequence to be analyzed; using the binocular camera image frame sequence to be analyzed and the binocular camera posture data as inputs of a pre-trained binocular asynchronous neural network, and outputting the 3D key point coordinates of each frame of gesture as a gesture tracking result; in the processing of two adjacent frames, the gesture tracking method will perform the same processing on a single camera image of a certain frame, so it only needs to be run once, and the result of the first pass is reused for the second time, which greatly reduces the calculation complexity, CPU occupancy and power consumption; shortens the processing time, can achieve a high frame rate inference speed, and realizes a fast and high-precision gesture tracking algorithm.

[0069] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0070] Embodiment 1:

[0071] like Figure 1 to Figure 3 As shown, the present invention provides a high frame rate gesture tracking method for a binocular camera, and the gesture tracking method comprises the following steps:

[0072] S1. Based on a preset binocular camera, obtaining the position and posture data of the first camera and the second camera;

[0073] S2, acquiring a binocular camera image frame sequence to be analyzed by alternately sampling images frame by frame through the first camera and the second camera;

[0074] S3. The binocular camera image frame sequence and the binocular camera pose data to be analyzed are used as the input of the pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result.

[0075] The gesture tracking method performs the same processing on a single camera image of a frame in the processing of two adjacent frames, so it only needs to be run once, and the result of the first pass is reused the second time, which greatly reduces the calculation complexity, CPU occupancy and power consumption; shortens the processing time, can achieve a high frame rate inference speed, and realizes a fast and high-precision gesture tracking algorithm.

[0076] like Figure 3 As shown, in S1, the binocular camera includes: a first camera, a second camera and a connecting device;

[0077] The first camera is fixedly connected to the second camera via a connecting device;

[0078] The first camera and the second camera are RGB cameras or grayscale cameras of the same model;

[0079] The first camera and the second camera simultaneously sample images in a hardware clock synchronization manner.

[0080] like Figure 2 As shown, in S1, the binocular camera posture data is reflected by a posture matrix, which is represented as a rotation quaternion and a three-dimensional translation vector.

[0081] In S2, the binocular camera image frame sequence to be analyzed is the first camera image and the second camera image alternating frame by frame, which is represented by the first camera image or the second camera image of the previous frame and the second camera image or the first camera image of the current frame; for example: the second frame input is the first camera image of the first frame and the second camera image of the second frame, the third frame input is the second camera image of the second frame and the first camera image of the third frame, the fourth frame input is the first camera image of the third frame and the second camera image of the fourth frame, and so on.

[0082] In S3, the binocular asynchronous neural network training includes the following steps:

[0083] Based on a preset binocular camera, obtaining the position and posture data of the first camera and the second camera;

[0084] The first camera and the second camera simultaneously sample images at the same frequency, and based on the image of each frame and the binocular camera posture data, the coordinates of the 3D key points of the gesture in each frame are obtained;

[0085] The first camera image and the second camera image are discarded frame by frame alternately from the images sampled simultaneously by the first camera and the second camera at the same frequency, to obtain multiple groups of binocular camera image frame sequences;

[0086] Each set of binocular camera image frame sequence and binocular camera pose data is used as a training sample, and the corresponding 3D key point coordinates of each frame of gesture are used as supervision signals to build a training database;

[0087] Network training is performed based on the training database to obtain a binocular asynchronous neural network.

[0088] like Figure 2 As shown, in S3, the binocular asynchronous neural network includes: an image encoding network, a pose projection network, a time alignment network, a feature fusion layer, a timing network and a decoding network;

[0089] The image encoding network adopts a convolutional neural network;

[0090] The pose projection network is used to map the 2D feature maps generated by the first camera and the second camera into a 3D space with the same origin, so that the hand poses inferred from the first camera image and the second camera image are at the same 3D position; the implementation method is to convert the pose relationship into a 4*4 pose matrix, perform channel-level multiplication with the feature map, and obtain a 3D pose projection;

[0091] The time alignment network is used to align the time dimension, aligning the pose inferred from the image of the previous frame to the current frame, and is a multi-layer perceptron structure;

[0092] The feature fusion layer is a 1*1 convolutional layer;

[0093] The timing network is an RNN recurrent neural network, which includes: one or more of a CNN module with a recurrent structure, an LSTM network, a GRU network, and a Transformer network;

[0094] The decoding network is a superposition of an attention layer and a multi-layer perceptron layer.

[0095] The binocular camera image frame sequence and binocular camera pose data to be analyzed are used as the input of the pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result, including the following steps:

[0096] The first camera image or the second camera image of the previous frame is converted into a feature map through the image encoding network. The feature map and the corresponding camera pose data are converted into a 3D pose projection through the pose projection network. The 3D pose projection is aligned to the pose processed by the previous frame image to the input feature fusion layer of the current frame through the time alignment network.

[0097] The second camera image or the first camera image of the current frame is processed by the image encoding network and the corresponding camera pose data is input into the pose projection network, and then input into the feature fusion layer after being processed by the pose projection network;

[0098] The feature fusion layer fuses the two input 3D posture projections into a unified 3D posture projection input timing network;

[0099] The temporal network stores the feature information after inference of each frame and transmits it to the next frame inference network as input to maintain the continuity of the previous and next frame data;

[0100] The decoding network decodes the feature data and outputs the 3D key point coordinates of each frame of gesture as the gesture tracking result.

[0101] In the processing of two adjacent frames in the gesture tracking method, the image coding network and the pose projection network will perform the same processing on a single camera image of a certain frame, so it only needs to be run once, and the result of the first pass is reused for the second time, which greatly reduces the computational complexity, CPU occupancy and power consumption; and the processing of all image coding networks and pose projection networks occupies 90% of the overall processing time, so this method shortens the processing time of a single frame image by nearly half, can achieve a high frame rate inference speed, and realizes a fast and high-precision gesture tracking algorithm.

[0102] Embodiment 2:

[0103] The present invention provides a binocular camera high frame rate gesture tracking system, the gesture tracking system comprising: a posture acquisition module, an image frame sequence acquisition module and a gesture tracking module;

[0104] The posture acquisition module is used to acquire the posture data of the first camera and the second camera;

[0105] The image frame sequence acquisition module is used to acquire a binocular camera image frame sequence to be analyzed by alternately sampling images frame by frame through the first camera and the second camera;

[0106] The gesture tracking module is used to use the binocular camera image frame sequence to be analyzed and the binocular camera posture data as the input of the pre-trained binocular asynchronous neural network, and output the 3D key point coordinates of each frame of gesture as the gesture tracking result.

[0107] The training of the binocular asynchronous neural network includes the following steps:

[0108] The first camera and the second camera simultaneously sample images at the same frequency, and based on the image of each frame and the binocular camera posture data, the coordinates of the 3D key points of the gesture in each frame are obtained;

[0109] The first camera image and the second camera image are discarded frame by frame alternately from the images sampled simultaneously by the first camera and the second camera at the same frequency, to obtain multiple groups of binocular camera image frame sequences;

[0110] Each set of binocular camera image frame sequence and binocular camera pose data is used as a training sample, and the corresponding 3D key point coordinates of each frame of gesture are used as supervision signals to build a training database;

[0111] Network training is performed based on the training database to obtain a binocular asynchronous neural network.

[0112] The binocular asynchronous neural network includes: an image encoding network, a posture projection network, a time alignment network, a feature fusion layer, a timing network and a decoding network;

[0113] The image encoding network adopts a convolutional neural network;

[0114] The time alignment network is a multi-layer perceptron structure;

[0115] The feature fusion layer is a convolutional layer;

[0116] The timing network is an RNN recurrent neural network, which includes: one or more of a CNN module with a recurrent structure, an LSTM network, a GRU network, and a Transformer network;

[0117] The decoding network is a superposition of an attention layer and a multi-layer perceptron layer.

[0118] The binocular camera image frame sequence and binocular camera pose data to be analyzed are used as the input of the pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result, including the following steps:

[0119] The first camera image or the second camera image of the previous frame is converted into a feature map through an image encoding network, and the feature map and the corresponding camera pose data are converted into a 3D pose projection through a pose projection network. The 3D pose projection is aligned to the pose processed by the previous frame image to the input feature fusion layer of the current frame through a time alignment network;

[0120] The second camera image or the first camera image of the current frame is processed by the image encoding network and the corresponding camera pose data is input into the pose projection network, and then input into the feature fusion layer after being processed by the pose projection network;

[0121] The feature fusion layer fuses the two input 3D posture projections into a unified 3D posture projection input timing network;

[0122] The temporal network stores the feature information after inference of each frame and transmits it to the next frame inference network as input to maintain the continuity of the previous and next frame data;

[0123] The decoding network decodes the feature data and outputs the coordinates of the 3D key points of each frame of gesture.

[0124] The binocular camera comprises: a first camera, a second camera and a connecting device;

[0125] The first camera is fixedly connected to the second camera via a connecting device;

[0126] The first camera and the second camera are RGB cameras or grayscale cameras of the same model.

[0127] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high frame rate gesture tracking method using a binocular camera, characterized in that: The gesture tracking method comprises the following steps: Based on a preset binocular camera, obtaining position and posture data of the first camera and the second camera; Acquire a binocular camera image frame sequence to be analyzed by alternately sampling images frame by frame with the first camera and the second camera; The binocular camera image frame sequence and binocular camera pose data to be analyzed are used as the input of the pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result.

2. The binocular camera high frame rate gesture tracking method according to claim 1, characterized in that: The training of the binocular asynchronous neural network includes the following steps: The first camera and the second camera simultaneously sample images at the same frequency, and based on the image of each frame and the binocular camera posture data, the coordinates of the 3D key points of the gesture in each frame are obtained; The first camera image and the second camera image are discarded frame by frame alternately from the images sampled simultaneously by the first camera and the second camera at the same frequency, to obtain multiple groups of binocular camera image frame sequences; Each set of binocular camera image frame sequence and binocular camera pose data is used as a training sample, and the corresponding 3D key point coordinates of each frame of gesture are used as supervision signals to build a training database; Network training is performed based on the training database to obtain a binocular asynchronous neural network.

3. The binocular camera high frame rate gesture tracking method according to claim 1, characterized in that: The binocular asynchronous neural network includes: an image encoding network, a posture projection network, a time alignment network, a feature fusion layer, a timing network and a decoding network; The image encoding network adopts a convolutional neural network; And / or the time alignment network is a multi-layer perceptron structure; And / or the feature fusion layer is a convolutional layer; And / or the timing network is an RNN recurrent neural network, and the RNN recurrent neural network includes: one or more of a CNN module with a recurrent structure, an LSTM network, a GRU network, and a Transformer network; And / or the decoding network is a superposition of an attention layer and a multi-layer perceptron layer.

4. The binocular camera high frame rate gesture tracking method as claimed in claim 3, characterized in that: The binocular camera image frame sequence and binocular camera pose data to be analyzed are used as the input of the pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result, including the following steps: The first camera image or the second camera image of the previous frame is converted into a feature map through an image encoding network, and the feature map and the corresponding camera pose data are converted into a 3D pose projection through a pose projection network. The 3D pose projection is aligned to the pose processed by the previous frame image to the input feature fusion layer of the current frame through a time alignment network; The second camera image or the first camera image of the current frame is processed by the image encoding network and the corresponding camera pose data is input into the pose projection network, and then input into the feature fusion layer after being processed by the pose projection network; The feature fusion layer fuses the two input 3D posture projections into a unified 3D posture projection input timing network; The temporal network stores the feature information after inference of each frame and transmits it to the next frame inference network as input to maintain the continuity of the previous and next frame data; The decoding network decodes the feature data and outputs the coordinates of the 3D key points of each frame of gesture.

5. The binocular camera high frame rate gesture tracking method according to claim 1, characterized in that: The binocular camera comprises: a first camera, a second camera and a connecting device; The first camera is fixedly connected to the second camera via a connecting device; The first camera and the second camera are RGB cameras or grayscale cameras of the same model.

6. A binocular camera high frame rate gesture tracking system, characterized in that: The gesture tracking system comprises: a posture acquisition module, an image frame sequence acquisition module and a gesture tracking module; The posture acquisition module is used to acquire the posture data of the first camera and the second camera; The image frame sequence acquisition module is used to acquire a binocular camera image frame sequence to be analyzed by alternately sampling images frame by frame through the first camera and the second camera; The gesture tracking module is used to use the binocular camera image frame sequence to be analyzed and the binocular camera posture data as the input of the pre-trained binocular asynchronous neural network, and output the 3D key point coordinates of each frame of gesture as the gesture tracking result.

7. The binocular camera high frame rate gesture tracking system according to claim 6, characterized in that: The training of the binocular asynchronous neural network includes the following steps: The first camera and the second camera simultaneously sample images at the same frequency, and based on the image of each frame and the binocular camera posture data, the coordinates of the 3D key points of the gesture in each frame are obtained; The first camera image and the second camera image are discarded frame by frame alternately from the images sampled simultaneously by the first camera and the second camera at the same frequency, to obtain multiple groups of binocular camera image frame sequences; Each set of binocular camera image frame sequence and binocular camera pose data is used as a training sample, and the corresponding 3D key point coordinates of each frame of gesture are used as supervision signals to build a training database; Network training is performed based on the training database to obtain a binocular asynchronous neural network.

8. The binocular camera high frame rate gesture tracking system as claimed in claim 6, characterized in that: The binocular asynchronous neural network includes: an image encoding network, a posture projection network, a time alignment network, a feature fusion layer, a timing network and a decoding network; The image encoding network adopts a convolutional neural network; And / or the time alignment network is a multi-layer perceptron structure; And / or the feature fusion layer is a convolutional layer; And / or the timing network is an RNN recurrent neural network, and the RNN recurrent neural network includes: one or more of a CNN module with a recurrent structure, an LSTM network, a GRU network, and a Transformer network; And / or the decoding network is a superposition of an attention layer and a multi-layer perceptron layer.

9. The binocular camera high frame rate gesture tracking system as claimed in claim 8, characterized in that: The binocular camera image frame sequence and binocular camera pose data to be analyzed are used as the input of the pre-trained binocular asynchronous neural network, and the 3D key point coordinates of each frame of gesture are output as the gesture tracking result, including the following steps: The first camera image or the second camera image of the previous frame is converted into a feature map through an image encoding network, and the feature map and the corresponding camera pose data are converted into a 3D pose projection through a pose projection network. The 3D pose projection is aligned to the pose processed by the previous frame image to the input feature fusion layer of the current frame through a time alignment network; The second camera image or the first camera image of the current frame is processed by the image encoding network and the corresponding camera pose data is input into the pose projection network, and then input into the feature fusion layer after being processed by the pose projection network; The feature fusion layer fuses the two input 3D posture projections into a unified 3D posture projection input timing network; The temporal network stores the feature information after inference of each frame and transmits it to the next frame inference network as input to maintain the continuity of the previous and next frame data; The decoding network decodes the feature data and outputs the coordinates of the 3D key points of each frame of gesture.

10. The binocular camera high frame rate gesture tracking system according to claim 6, characterized in that: The binocular camera comprises: a first camera, a second camera and a connecting device; The first camera is fixedly connected to the second camera via a connecting device; The first camera and the second camera are RGB cameras or grayscale cameras of the same model.

Images