Bimodal gesture recognition method and device in complex scene

Abstract

The invention discloses a bimodal gesture recognition method and device in a complex scene, and the method comprises the steps: synchronously collecting an RGB image sequence and a plurality of event frames, obtaining a hand region prediction frame corresponding to each event frame based on the collected RGB image sequence and the plurality of event frames, a preset target detection network, and a preset hand frame prediction model, the hand region prediction frame comprises a first type of hand region prediction frame and a second type of hand region prediction frame; and inputting the synchronously acquired RGB image sequence, the plurality of event frames and the corresponding hand region prediction frame into the fusion skeleton point extraction network to obtain gesture skeleton point information of each frame, and feeding back and inputting the gesture skeleton point information of each frame into the hand region prediction model. According to the bimodal gesture recognition method and device in the complex scene, real-time recognition is carried out on the scene where dynamic and static gestures appear in a mixed mode, and the recognition accuracy is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a dual-modal gesture recognition method and device in complex scenarios. Background Art

[0002] In the field of computer image gesture recognition, obtaining skeleton points from RGB images and performing gesture recognition has been able to achieve a high recognition accuracy. However, there are still some problems in this process. One of the main problems is that there are strict requirements on the quality of the input RGB image. In the face of underexposure or overexposure caused by lighting conditions such as darkness and backlight, or motion blur caused by the fast speed of dynamic gestures, algorithms based on RGB image input often find it difficult to obtain satisfactory results. Most of the existing dynamic and static gesture recognition methods first perform static recognition on each frame of graphics separately, and then reprocess the sequence obtained by concatenating static graphics to obtain the corresponding dynamic gesture. However, not all states in the process of dynamic gesture movement are suitable for representation by static gestures. It is easy to recognize wrong static gestures during the process of dynamic gesture movement, thereby affecting the recognition accuracy of the final dynamic gesture. For example, in the dynamic gesture "swipe left", the entire palm may be blocked by the wrist at some moments. At this time, it is not appropriate to perform static gesture recognition. In addition, this method will give dynamic and static gestures at the same time, which is easy to confuse the two. For example, when the static gesture "number 0" and the static gesture "number 5" are displayed alternately, they are easily mistakenly recognized as dynamic gestures of "zoom in" and "zoom out" during the switching process. Summary of the invention

[0003] In order to solve the technical problems existing in the background technology, the present invention proposes a dual-modal gesture recognition method and device in complex scenarios.

[0004] The present invention proposes a dual-modal gesture recognition method in a complex scenario, comprising the following steps:

[0005] S1, synchronously collecting an RGB image sequence and a plurality of event frames, and obtaining a hand region prediction frame corresponding to each event frame based on the collected RGB image sequence and the plurality of event frames, a preset target detection network, and a preset hand region prediction model, wherein the hand region prediction frame includes a first type hand region prediction frame and a second type hand region prediction frame;

[0006] S2, inputting the synchronously collected RGB image sequence, multiple event frames and the corresponding hand area prediction frame into the fusion skeleton point extraction network to obtain the gesture skeleton point information of each frame, and feeding back the gesture skeleton point information of each frame into the hand frame prediction model;

[0007] S3, continuously inputting gesture skeleton point information corresponding to a plurality of event frames one by one into a skeleton point windowing algorithm according to the time sequence of the plurality of event frames to obtain a fixed-length skeleton point sequence;

[0008] S4, inputting the fixed-length skeleton point sequence into the adaptive graph convolutional neural network in sequence to obtain the corresponding gesture classification;

[0009] S5. The gesture classification input and output correction module is used to confirm the gesture state classification to obtain a gesture recognition result for output.

[0010] Preferably, step S1 specifically includes:

[0011] S11, synchronously collecting RGB image sequence and multiple event frames;

[0012] S12, inputting the collected RGB image sequence and multiple event frames into the target detection network to obtain a first type of hand region prediction frame;

[0013] S13, inputting the gesture skeleton point information at the previous moment, the RGB image sequence collected at the current moment, and multiple event frames into a preset hand frame prediction model to obtain a corresponding second type hand area prediction frame.

[0014] Preferably, the object detection network includes a feature extraction module, a feature fusion module and a bounding box predictor; step S12 specifically includes:

[0015] S121, inputting the collected RGB image sequence and multiple event frames into a feature extraction module to obtain corresponding RGB image feature vectors and event feature vectors;

[0016] S122, inputting the RGB image feature vector and the event feature vector into a feature fusion module to obtain the coordinates and size of the hand region prediction box;

[0017] S123: Input the coordinates and size of the hand region prediction box into a bounding box predictor to obtain a first type hand region prediction box.

[0018] Preferably, the hand frame prediction model includes a loop control system and a Kalman filter; step S13 specifically includes:

[0019] S131, obtaining the gesture skeleton point information of the previous moment output by the fusion skeleton point extraction network through the loop control system, and inputting the gesture skeleton point information of the previous moment into the hand frame prediction network;

[0020] S132, inputting the gesture skeleton point information of the previous moment, the RGB image sequence corresponding to the current moment, and multiple event frames into the Kalman filter to obtain the second type hand area prediction frame corresponding to the current moment.

[0021] Preferably, the fusion skeleton point extraction network includes an encoder module and a decoder module; step S2 specifically includes:

[0022] S21, inputting the collected RGB image sequence and multiple event frames into the encoder module frame by frame to extract corresponding multiple deep semantic features;

[0023] S22, inputting multiple deep semantic features into the decoder module to obtain gesture skeleton point information corresponding to multiple event frames one by one, and feeding back the gesture skeleton point information of the current frame into the hand frame prediction model.

[0024] Preferably, the adaptive graph convolutional neural network includes a batch normalization layer, a plurality of adaptive graph convolution blocks, a global average pooling layer and a normalized index layer; step S4 specifically includes:

[0025] S41, inputting the fixed-length skeleton point sequence into the batch normalization layer for batch normalization processing, and then sequentially passing through multiple adaptive graph convolution blocks, a global average pooling layer and a normalized index layer to obtain a one-dimensional vector of length x, i.e., the adaptive graph convolution network classification probability;

[0026] S42, classifying the adaptive graph convolution network with the highest classification probability score as the gesture state;

[0027] Wherein, x is the number of set gesture categories.

[0028] Preferably, the adaptive graph convolutional neural network training process is as follows:

[0029] Set x types of gesture classification, and establish corresponding relationships between multiple groups of skeleton point sequences used for training and the set x types of gesture classification to form a training sample set;

[0030] Taking the trained skeleton point sequence as input and the gesture classification corresponding to the trained skeleton point sequence as output, the preset adaptive graph convolutional neural network is supervised trained based on the training sample set to obtain a trained adaptive graph convolutional neural network.

[0031] The present invention proposes a dual-modal gesture recognition device in a complex scenario, comprising:

[0032] A hand region prediction module, used for synchronously collecting an RGB image sequence and a plurality of event frames, and obtaining a hand region prediction frame corresponding to each event frame based on the collected RGB image sequence and the plurality of event frames, a preset target detection network, and a preset hand frame prediction model, wherein the hand region prediction frame includes a first type hand region prediction frame and a second type hand region prediction frame;

[0033] An information extraction module is used to input the synchronously collected RGB image sequence, multiple event frames and the corresponding hand area prediction frame into the fusion skeleton point extraction network to obtain the gesture skeleton point information of each frame, and feed back the gesture skeleton point information of each frame into the hand frame prediction model;

[0034] An information processing module, used for continuously inputting gesture skeleton point information corresponding to a plurality of event frames one by one into a skeleton point windowing algorithm in a time sequence of the plurality of event frames to obtain a fixed-length skeleton point sequence;

[0035] The gesture state classification module is used to input a fixed-length skeleton point sequence into an adaptive graph convolutional neural network to obtain the corresponding gesture classification;

[0036] The gesture recognition output module is used to input the gesture classification into the output correction module to confirm the gesture state classification so as to obtain the gesture recognition result for output.

[0037] In the present invention, the proposed dual-modal gesture recognition method and device in complex scenes, for mixed dynamic and static gesture actions, respectively collects RGB image sequences and event streams by calibrating and synchronizing traditional RGB cameras and event cameras, and then fuses the skeleton point extraction algorithm optimized by dual-modal input loop control to obtain the skeleton point sequence, and uses the adaptive graph convolution network trained for the dynamic and static gesture joint real-time recognition scene to perform real-time recognition of the gesture skeleton point sequence, and finally uses the output correction module to control the network recognition result to obtain the final output result. Real-time recognition is performed for scenes where dynamic and static gestures appear mixed, ensuring the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the distribution structure of acquisition devices for a dual-modal gesture recognition method in a complex scenario proposed by the present invention;

[0039] Figure 2 This is an overall flow chart of a method for real-time joint recognition of dynamic and static gestures in a dual-modal gesture recognition method in complex scenarios proposed by the present invention;

[0040] Figure 3 A dual-modal hand target detection network framework diagram of a dual-modal gesture recognition method in complex scenarios proposed by the present invention;

[0041] Figure 4 This is a fusion skeleton point extraction network framework diagram of a dual-modal gesture recognition method in complex scenarios proposed by the present invention;

[0042] Figure 5 This is a diagram of an adaptive graph convolutional neural network framework for a dual-modal gesture recognition method in complex scenarios proposed by the present invention;

[0043] Figure 6 This is a flow chart of the control output of the recognition result by the output correction module of the dual-modal gesture recognition method in complex scenarios proposed by the present invention;

[0044] Figure 7 A schematic diagram of the work flow structure of a dual-modal gesture recognition method in a complex scenario proposed by the present invention;

[0045] Figure 8 This is a schematic diagram of the system architecture of a dual-modal gesture recognition device in a complex scenario proposed by the present invention. DETAILED DESCRIPTION

[0046] Reference Figure 1-8 The present invention proposes a dual-modal gesture recognition method in a complex scene, comprising the following steps:

[0047] S1. Synchronously collect an RGB image sequence and multiple event frames, and obtain a hand area prediction frame corresponding to each event frame based on the collected RGB image sequence and multiple event frames, a preset target detection network, and a preset hand frame prediction model. The hand area prediction frame includes a first type of hand area prediction frame and a second type of hand area prediction frame.

[0048] In this embodiment, an input image resolution of 640×480, a traditional RGB camera sampling frame rate of 30 FPS, each RGB image frame corresponds to 9 event frames, 21 skeleton points are used to represent gestures, and 29 gesture actions are recognized for illustration.

[0049] In this embodiment, a set of registered RGB cameras and event cameras are used as acquisition devices to obtain spatiotemporally synchronized RGB images and event information, such as Figure 1 The processing module is a high-performance AI edge device with a TPU (Tensor Processing Unit), which can realize the deployment of dual-modal gesture recognition algorithms and support various stage algorithms including target detection network, hand area selection algorithm, skeleton point extraction network, skeleton point windowing algorithm, and adaptive graph convolutional neural network.

[0050] In this embodiment, step S1 specifically includes:

[0051] S11. Synchronously collect an RGB image sequence and multiple event frames.

[0052] In this embodiment, the synchronization between the RGB traditional camera and the event camera is divided into two aspects: time synchronization and space synchronization, which are as follows:

[0053] 1. Time synchronization: Use a synchronization line to connect the traditional RGB camera and the event camera so that a unified timestamp can be used.

[0054] 2. Spatial synchronization: Use a beam splitter to split the light in the same field of view into two beams, which are collected by the RGB traditional camera and the event camera respectively. After the hardware is set up, the image is spatially calibrated by using a flashing checkerboard image (required by the event camera characteristics). The image obtained by the camera on one side reflected by the beam splitter needs to be flipped left and right to be consistent with the image of the camera on the refracting side.

[0055] For each frame of RGB image collected, the event stream is evenly compressed using the time surface method to obtain 9 frames of event images, which are then aligned with the RGB image in the time dimension as subsequent event-side input.

[0056] S12. Input the collected RGB image sequence and multiple event frames into the target detection network to obtain a first type of hand region prediction frame.

[0057] In this embodiment, the target detection network includes a feature extraction module, a feature fusion module and a bounding box predictor; step S12 specifically includes:

[0058] S121, inputting the collected RGB image sequence and multiple event frames into a feature extraction module to obtain corresponding RGB image feature vectors and event feature vectors;

[0059] S122, inputting the RGB image feature vector and the event feature vector into a feature fusion module to obtain the coordinates and size of the hand region prediction box;

[0060] S123: Input the coordinates and size of the hand region prediction box into a bounding box predictor to obtain a first type hand region prediction box.

[0061] S13, inputting the gesture skeleton point information at the previous moment, the RGB image sequence collected at the current moment, and multiple event frames into a preset hand frame prediction model to obtain a corresponding second type hand area prediction frame.

[0062] In this embodiment, the image sequence and event stream collected by the electronic device are as follows: Figure 2 As shown in the first half of the skeleton point extraction algorithm, the target detection network and the hand area selection algorithm are used in turn to obtain the hand selection area for subsequent skeleton point extraction network processing.

[0063] The image sequence and event stream collected by the electronic device are passed through Figure 2 The object detection network and hand region selection algorithm in the image are used to obtain the hand selection region for subsequent skeleton point extraction network to process.

[0064] Among them, the overall block diagram of the target detection network is as follows Figure 3 As shown in (a), it consists of 3 sets of target detection feature extraction modules, 3 sets of target detection feature fusion modules and 1 bounding box predictor. Each feature extraction module consists of 2 3×3 convolutional layers, 2 linear rectified linear units (ReLU) layers and 1 2×2 maximum pooling layer. Figure 3 (b) is shown. The first-stage feature extractor expands the number of channels of the RGB three-channel image and the matching 9-channel event image to 64 respectively, and reduces its resolution to 320×240 (written as 320×240×64, similar expressions in the following text have the same meaning), which is called the first-stage extracted features of target detection (referred to as the first extracted features when introducing the target detection network, the same below); the subsequent second-stage and third-stage feature extractors repeat the above operations to obtain the second extracted features of 160×120×128 and the third extracted features of 80×60×256 respectively. The target detection network extracts deep semantic information from the image through the above three-stage feature extraction module.

[0065] Each set of feature fusion modules contains one spatial attention fusion module and one channel attention fusion module, such as Figure 3(c) is shown. In the first-stage feature fusion module, the third extracted features with dimensions of 80×60×256 from the RGB side and the event side are first fused with spatial attention. After point addition, the third extracted features are respectively obtained as 80×60×1 vectors by channel maximum pooling (MaxPooling) and channel average pooling (Average Pooling). After concatenation, the two are respectively obtained as 80×60×1 spatial attention weights by 1×1 convolution and Sigmoid activation layer, and are acted on the original third extracted features by point multiplication, thereby outputting intermediate fusion features with a size of 80×60×256. This process realizes the exchange and complementarity of messages between different modalities, using the rich motion information of the event camera under various lighting conditions to supplement the difficulties of the traditional RGB camera in facing high-speed targets and harsh lighting environments, and using the rich texture details of the traditional RGB camera to supplement the problem that the event camera is difficult to capture for static targets. Subsequently, the above intermediate fusion features will be further fused with channel attention. The results of this part of the network are basically the same as those of spatial attention fusion, except that the pooling step is shifted from the channel dimension to the spatial dimension. Spatial maximum pooling and spatial average pooling are used to obtain a 1×1×256 vector and processed into a 1×1×256 channel attention weight to act on the intermediate fusion feature, thereby obtaining an 80×60×256 first-stage fusion feature for target detection (referred to as the first fusion feature when introducing the target detection network, the same below). When the spatial attention weight and channel attention weight play their roles, the former is multiplied pixel by pixel with the feature map of each channel in the third extracted feature, and the latter multiplies the feature map of each channel of the intermediate fusion feature by the weight of the channel as a whole. The subsequent first fusion feature will continue to pass through the second and third stage feature fusion modules, and finally obtain the third fusion module to be sent to the bounding box predictor.

[0066] The final bounding box predictor is essentially composed of a fully connected layer, which uses each pixel in the 80×60×256 third fusion feature to calculate the coordinates and size of the hand area prediction box.

[0067] It should be noted that the above target detection network is not used for every frame. After the hand region prediction frame is given in the first frame and sent to the hand frame prediction algorithm, the target detection network enters a dormant state, and the hand region frame information of subsequent frames is given by the hand frame prediction algorithm. The hand frame prediction algorithm combines the results of the skeleton point extraction network of the previous frame, introduces a loop control system, and uses the Kalman filter to process the process as follows:

[0068]

[0069] The coordinates of the skeleton points of the current frame are predicted, and the prediction results are appropriately enlarged and directly input into the skeleton point extraction network as the hand area prediction frame for calculation. Where I represents the unit matrix, represents the posterior estimate of the current state at time k. In this method, the posterior estimate is the coordinate of the gesture skeleton point, z k is the measurement value of the current state (skeleton point in this method), is the state estimation covariance, A k is the state transfer matrix, B k u k The term is used to simulate the effect of the external environment, H k is the state observation matrix, Q k With R k The terms represent the covariance of process noise and measurement noise, respectively. k is the Kalman coefficient, which is used to represent the measured value z k For and Item, from the previous frame state and The predicted values ​​are the prior state estimates, which are respectively and Indicates that, and with the introduction of the measured value z k The final result is the optimal estimate of the current state, which is also the posterior state estimate.

[0070] Generally speaking, when processing each frame of data, you need to get a priori estimate With the measured value z k , and then use the posterior estimate As output. However, in this method, Figure 1 As can be seen from the figure, a prediction box needs to be given first to obtain the measurement value from the skeleton point extraction network, so this method directly uses the prior estimate As the output of the hand region box prediction algorithm, the measurement value z is obtained from the skeleton point extraction network k Update the posterior estimate Used to continue calculating the prior estimate for the next frame

[0071] In this embodiment, when the skeleton point extraction network cannot obtain the skeleton point information from the hand area prediction frame generated by the Kalman filter, the target detection network is re-enabled to correct the hand area prediction frame position of the current value. In this way, the method can make full use of the continuous spatial information in the RGB video stream and the event stream to optimize the prediction of the hand area, avoid the continuous use of the target detection network with large computational complexity and long time consumption, and use the traditional Kalman filter to obtain an approximate effect.

[0072] In this embodiment, the hand frame prediction model includes a loop control system and a Kalman filter; step S13 specifically includes:

[0073] S131, obtaining the gesture skeleton point information of the previous moment output by the fusion skeleton point extraction network through the loop control system, and inputting the gesture skeleton point information of the previous moment into the hand frame prediction network;

[0074] S132, inputting the gesture skeleton point information of the previous moment, the RGB image sequence corresponding to the current moment, and multiple event frames into the Kalman filter to obtain the second type hand area prediction frame corresponding to the current moment.

[0075] S2. Input the synchronously collected RGB image sequence, multiple event frames and the corresponding hand area prediction frame into the fusion skeleton point extraction network to obtain the gesture skeleton point information of each frame, and feed back the gesture skeleton point information of each frame into the hand frame prediction model.

[0076] In this embodiment, the fusion skeleton point extraction network includes an encoder module and a decoder module; step S2 specifically includes:

[0077] S21, inputting the collected RGB image sequence and multiple event frames into the encoder module frame by frame to extract corresponding multiple deep semantic features;

[0078] S22, inputting multiple deep semantic features into the decoder module to obtain gesture skeleton point information corresponding to multiple event frames one by one, and feeding back the gesture skeleton point information of the current frame into the hand frame prediction model.

[0079] In this embodiment, if Figure 4As shown in the figure, the fusion skeleton point extraction network adopts a symmetrical U-shaped structure as a whole. The first half is called the encoder, which downsamples the image and extracts the deep semantic features in the image; the second half is called the decoder, which upsamples the image and obtains the gesture skeleton point information from the deep semantic features. The encoder consists of three feature extraction modules and three groups of feature fusion modules. The structural design of the feature extraction module and the feature fusion module is consistent with that in the target detection network, and the difference lies in their arrangement. In order to obtain more accurate skeleton point coordinates, the network needs to consider more global coordinate information, that is, it needs to retain more shallow semantic features. Therefore, the encoder alternately distributes the feature extraction module and the feature fusion module so that the RGB side and the event side information can be exchanged and supplemented at a shallower level. In contrast, the decoder consists of three decoding modules and one 1×1 convolution layer. Each decoding module contains two 3×3 convolutions and one 2×2 deconvolution (also known as transposed convolution). The decoding modules at each stage reduce the number of channels to 128, 64, and 64, respectively, and double the resolution in length and width to 160×120, 320×240, and 640×480. Since each modality has been fully feature extracted in the encoder, only one fused signal is retained in the decoder instead of two signals in the encoder.

[0080] In addition, the second and third decoding modules will be connected to the corresponding modules in the encoder through skip connections (dotted lines in the figure). For example, the two 320×240×128 signals output by the encoder in the first stage and the 320×240×128 signals output by the decoder in the second stage are spliced ​​into the third stage of 320×240×384. In other words, the two semantic features output by the feature fusion module of each stage of the encoder are copied and spliced ​​with the corresponding semantic feature vectors in the decoder. In this way, the semantic information in the deep features (i.e., the upsampled part) is fully utilized, and the spatial information in the shallow features (i.e., the part obtained by the skip connection) is retained, so as to better complete the skeleton point extraction task.

[0081] The 1×1 convolution at the end of the decoder further reduces the original 64 channels to 21 channels. Each channel represents the probability heat map of each bone point. The coordinates of the bone point corresponding to the channel can be determined by determining the position where the maximum probability occurs.

[0082] S3. Continuously inputting gesture skeleton point information corresponding to a plurality of event frames one by one into a skeleton point windowing algorithm in a time sequence of the plurality of event frames to obtain a fixed-length skeleton point sequence.

[0083] S4. Input the fixed-length skeleton point sequence into the adaptive graph convolutional neural network in sequence to obtain the corresponding gesture classification.

[0084] In this embodiment, the adaptive graph convolutional neural network includes a batch normalization layer, a plurality of adaptive graph convolution blocks, a global average pooling layer and a normalized index layer; step S4 specifically includes:

[0085] S41, inputting the fixed-length skeleton point sequence into the batch normalization layer for batch normalization processing, and then sequentially passing through multiple adaptive graph convolution blocks, a global average pooling layer and a normalized index layer to obtain a one-dimensional vector of length x, i.e., the adaptive graph convolution network classification probability;

[0086] S42, classifying the adaptive graph convolution network with the highest classification probability score as the gesture state;

[0087] Wherein, x is the number of set gesture categories.

[0088] In this embodiment, the adaptive graph convolutional neural network training process is as follows: x types of gesture classifications are set, and a correspondence is established between multiple groups of skeleton point sequences used for training and the set x types of gesture classifications to form a training sample set; the trained skeleton point sequence is used as input, and the gesture classification corresponding to the trained skeleton point sequence is used as output, and the preset adaptive graph convolutional neural network is supervised and trained based on the training sample set to obtain a trained adaptive graph convolutional neural network.

[0089] In this embodiment, the method uses the correspondence between the skeleton point sequence in the data set and the set 29 types of gestures to train an adaptive graph convolutional neural network. For each training sample, the one-dimensional vector of length 29 given by the network is compared with the actual gesture category of the sample, and the supervised network obtains the weight information in each structure, thereby realizing gesture recognition for the skeleton point sequence.

[0090] In this embodiment, the above skeleton point sequence is processed by an adaptive graph convolutional neural network to perform real-time joint recognition of mixed input dynamic and static gestures. Figure 2As shown in the figure, before the electronic device uses the adaptive graph convolutional neural network for real-time joint recognition of dynamic and static gestures, it is necessary to accumulate the continuously input bone coordinates to form a fixed-length bone point sequence, which is one of the signs to distinguish whether the algorithm can run in real time. For algorithms that do not require real-time operation, the recorded data is usually sent to the network for direct recognition, which can ensure that the input sequence contains a complete set of dynamic gestures or a single-frame static gesture; when the algorithm is running in real time, it is not possible to know the duration of the gesture in the data stream in advance, so it is necessary to set the sequence length in advance, and recognize the sequences within this period in sequence in a queue-like manner. The window length setting needs to be consistent with the sequence length set when the data set is made. Due to the different speeds of gestures of different objects, it may happen that the time used to complete the gesture is greater than or less than the set number of frames. Therefore, time enhancement is introduced when making the data set. By accelerating and decelerating and changing the time position, it is ensured that gestures at different speeds can be correctly recognized by the algorithm at different time stages.

[0091] In this embodiment, the adaptive graph convolutional network processes the input skeleton point sequence as follows: Figure 5 As shown in (a), after batch normalization, the skeleton point sequence of size 3 (number of channels) × 11 (number of sequence frames) × 21 (number of skeleton points) passes through 9 adaptive graph convolution blocks (referred to as graph convolution blocks in the figure) in sequence, and the number of channels becomes 64, 64, 64, 128, 128, 128, 256, 256, 256, and finally passes through the global average pooling (GAP) layer and the normalized index (Softmax) layer to obtain a one-dimensional vector of length 29, i.e., the classification probability of the adaptive graph convolution network. The one with the highest probability score is considered to be the classification result of the adaptive graph convolution network, and the score value is its confidence level.

[0092] In this embodiment, each adaptive graph convolution block in the adaptive graph convolution network is as follows: Figure 5 As shown in (b), it consists of an adaptive graph convolution layer, a batch normalization (BN) layer, a linear rectification layer, and a random dropout layer. Among them, the adaptive graph convolution layer plays a key role in feature extraction. Figure 5 As shown in (c). The three dashed boxes in the figure indicate the same structure, with only some parameters set differently, which will be discussed in detail later. Except for the residual connection, the corresponding formula of the adaptive graph convolution layer is as follows:

[0093]

[0094] Among them, taking the first graph convolution layer as an example, f in and f outRepresents the skeleton point sequence of the input network and the output features of the first graph convolution layer, whose sizes are C in (number of input channels, 3) × T (number of sequence frames, 11) × N (number of bone points, 21) and C out (number of output channels, 64)×T×N; A k , B k and C k are three adjacency matrices of size N×N; W k is a 1×1 convolution kernel, and the three constitute the basic form of graph convolution; K v is the number of subset categories, here it is 3.

[0095] In this embodiment, the specific process of obtaining the three adjacency matrices is as follows: k The adjacency matrix is ​​manually set, which is obtained based on the connection relationship between N skeleton points. In this method, (K v =3), namely (a) the current point and the adjacent points are set to the same weight; (b) the current point and the adjacent points use two different weights; (c) the current point, the adjacent points close to the hand center, and the adjacent points far from the hand center use three different weights. k is the adjacency matrix obtained completely by training. k In addition to using the trained parameters, Gaussian embedding is also used to play a role similar to the self-attention mechanism, which can be obtained through the formula as follows:

[0096] C k =softmax((W θk f in ) T W φk f in );

[0097] Among them, W θk With W φk are the parameters obtained through training.

[0098] S5. The gesture classification input and output correction module is used to confirm the gesture state classification to obtain a gesture recognition result for output.

[0099] In this embodiment, after the adaptive graph convolution network classification results are obtained through several adaptive graph convolution blocks and normalized index layers, in order to avoid misjudgment, it is necessary to further process it through the output correction module. Now, taking the threshold M=5 required for confirming gestures and the threshold N=3 required for denying gestures as an example, the process of the output correction module processing the gesture "hello" is shown, as shown in the figure. Figure 6As shown. At the beginning, the output correction module is in an idle state and outputs the "no gesture" state. When the classification result of the adaptive graph convolutional network is received as the gesture "hello", the output correction module sets the confirmation cumulative value m to 1, sets the denial cumulative value n to 0, and enters the preparation state, at which time the "confirming" state is output. Subsequently, when other results are received except the gesture "hello", the output correction module will accumulate the denial cumulative value n until it reaches the threshold N, and then return to the initial idle state; if the gesture "hello" is received, n is reduced by 1, and the confirmation cumulative value m is accumulated. When m reaches the threshold M, the output correction module will determine that the gesture is indeed "hello" at this time, jump to the confirmation state and output the gesture "hello". In this state, the reception of the gesture "hello" or other gestures will still act on the denial cumulative value n. When n reaches the threshold N, the output correction module determines that the gesture "hello" action has been completed, exits the confirmation state and returns to the idle state.

[0100] Reference Figure 1-8 The present invention proposes a dual-modal gesture recognition device in a complex scene, comprising:

[0101] A hand region prediction module is used to synchronously collect an RGB image sequence and multiple event frames, and obtain a hand region prediction frame corresponding to each event frame based on the collected RGB image sequence and multiple event frames, a preset target detection network, and a preset hand frame prediction model. The hand region prediction frame includes a first type of hand region prediction frame and a second type of hand region prediction frame.

[0102] An information extraction module is used to input the synchronously collected RGB image sequence, multiple event frames and the corresponding hand area prediction frame into the fusion skeleton point extraction network to obtain the gesture skeleton point information of each frame, and feed back the gesture skeleton point information of each frame into the hand frame prediction model;

[0103] An information processing module, used for continuously inputting gesture skeleton point information corresponding to a plurality of event frames one by one into a skeleton point windowing algorithm in a time sequence of the plurality of event frames to obtain a fixed-length skeleton point sequence;

[0104] The gesture state classification module is used to input a fixed-length skeleton point sequence into an adaptive graph convolutional neural network to obtain the corresponding gesture classification;

[0105] The gesture recognition output module is used to input the gesture classification into the output correction module to confirm the gesture state classification so as to obtain the gesture recognition result for output.

[0106] In the specific working process of the dual-modal gesture recognition method and device in complex scenes of this embodiment, for mixed dynamic and static gesture actions, the RGB image sequence and the event stream are respectively collected by calibrating and synchronizing the traditional RGB camera and the event camera, and then the skeleton point extraction algorithm optimized by the dual-modal input loop control is integrated to obtain the skeleton point sequence, and the adaptive graph convolution network trained for the dynamic and static gesture joint real-time recognition scene is used to perform real-time recognition of the gesture skeleton point sequence, and finally the output correction module is used to control the network recognition result to obtain the final output result.

[0107] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A dual-modal gesture recognition method in complex scenarios, characterized in that: The following steps are involved: S1, synchronously collecting an RGB image sequence and a plurality of event frames, and obtaining a hand region prediction frame corresponding to each event frame based on the collected RGB image sequence and the plurality of event frames, a preset target detection network, and a preset hand region prediction model, wherein the hand region prediction frame includes a first type hand region prediction frame and a second type hand region prediction frame; S2, inputting the synchronously collected RGB image sequence, multiple event frames and the corresponding hand area prediction frame into the fusion skeleton point extraction network to obtain the gesture skeleton point information of each frame, and feeding back the gesture skeleton point information of each frame into the hand frame prediction model; S3, continuously inputting gesture skeleton point information corresponding to a plurality of event frames one by one into a skeleton point windowing algorithm according to the time sequence of the plurality of event frames to obtain a fixed-length skeleton point sequence; S4, inputting the fixed-length skeleton point sequence into the adaptive graph convolutional neural network in sequence to obtain the corresponding gesture classification; S5. The gesture classification input and output correction module is used to confirm the gesture state classification to obtain a gesture recognition result for output.

2. The dual-modal gesture recognition method in complex scenes according to claim 1, characterized in that: Step S1 specifically includes: S11, synchronously collecting RGB image sequence and multiple event frames; S12, inputting the collected RGB image sequence and multiple event frames into the target detection network to obtain a first type of hand region prediction frame; S13, inputting the gesture skeleton point information at the previous moment, the RGB image sequence collected at the current moment, and multiple event frames into a preset hand frame prediction model to obtain a corresponding second type hand area prediction frame.

3. The dual-modal gesture recognition method in complex scenes according to claim 2, characterized in that: The target detection network includes a feature extraction module, a feature fusion module and a bounding box predictor; step S12 specifically includes: S121, inputting the collected RGB image sequence and multiple event frames into a feature extraction module to obtain corresponding RGB image feature vectors and event feature vectors; S122, inputting the RGB image feature vector and the event feature vector into a feature fusion module to obtain the coordinates and size of the hand region prediction box; S123: Input the coordinates and size of the hand region prediction box into a bounding box predictor to obtain a first type hand region prediction box.

4. The dual-modal gesture recognition method in complex scenes according to claim 3, characterized in that: The hand frame prediction model includes a loop control system and a Kalman filter; step S13 specifically includes: S131, obtaining the gesture skeleton point information of the previous moment output by the fusion skeleton point extraction network through the loop control system, and inputting the gesture skeleton point information of the previous moment into the hand frame prediction network; S132, inputting the gesture skeleton point information of the previous moment, the RGB image sequence corresponding to the current moment, and multiple event frames into the Kalman filter to obtain the second type hand area prediction frame corresponding to the current moment.

5. The dual-modal gesture recognition method in complex scenes according to claim 4, characterized in that: The fusion skeleton point extraction network includes an encoder module and a decoder module; Step S2 specifically includes: S21, inputting the collected RGB image sequence and multiple event frames into the encoder module frame by frame to extract corresponding multiple deep semantic features; S22, inputting multiple deep semantic features into the decoder module to obtain gesture skeleton point information corresponding to multiple event frames one by one, and feeding back the gesture skeleton point information of the current frame into the hand frame prediction model.

6. The dual-modal gesture recognition method in complex scenes according to claim 1, characterized in that: The adaptive graph convolutional neural network includes a batch normalization layer, a plurality of adaptive graph convolution blocks, a global average pooling layer and a normalized index layer; Step S4 specifically includes: S41, inputting the fixed-length skeleton point sequence into the batch normalization layer for batch normalization processing, and then sequentially passing through multiple adaptive graph convolution blocks, a global average pooling layer and a normalized index layer to obtain a one-dimensional vector of length x, i.e., the adaptive graph convolution network classification probability; S42, classifying the adaptive graph convolution network with the highest classification probability score as the gesture state; Wherein, x is the number of set gesture categories.

7. The dual-modal gesture recognition method in complex scenes according to claim 6, characterized in that: The adaptive graph convolutional neural network training process is as follows: Set x types of gesture classification, and establish corresponding relationships between multiple groups of skeleton point sequences used for training and the set x types of gesture classification to form a training sample set; Taking the trained skeleton point sequence as input and the gesture classification corresponding to the trained skeleton point sequence as output, the preset adaptive graph convolutional neural network is supervised trained based on the training sample set to obtain a trained adaptive graph convolutional neural network.

8. A dual-modal gesture recognition device in complex scenarios, characterized in that: include: A hand region prediction module is used to synchronously collect an RGB image sequence and multiple event frames, and obtain a hand region prediction frame corresponding to each event frame based on the collected RGB image sequence and multiple event frames, a preset target detection network, and a preset hand frame prediction model, wherein the hand region prediction frame includes a first type hand region prediction frame and a second type hand region prediction frame; An information extraction module is used to input the synchronously collected RGB image sequence, multiple event frames and the corresponding hand area prediction frame into the fusion skeleton point extraction network to obtain the gesture skeleton point information of each frame, and feed back the gesture skeleton point information of each frame into the hand frame prediction model; An information processing module, used for continuously inputting gesture skeleton point information corresponding to a plurality of event frames one by one into a skeleton point windowing algorithm in a time sequence of the plurality of event frames to obtain a fixed-length skeleton point sequence; The gesture state classification module is used to input a fixed-length skeleton point sequence into an adaptive graph convolutional neural network to obtain the corresponding gesture classification; The gesture recognition output module is used to input the gesture classification into the output correction module to confirm the gesture state classification so as to obtain the gesture recognition result for output.

Images