Gesture recognition method and system, computer equipment and readable storage medium

Abstract

The invention discloses a gesture recognition method and system, computer equipment and a readable storage medium, and relates to the technical field of image recognition. The method comprises the following steps: acquiring a to-be-recognized gesture image; and preprocessing the gesture image to be recognized according to the target detection algorithm, the single Gaussian model and the median filtering algorithm to obtain the preprocessed gesture image to be recognized. And performing feature extraction on the preprocessed to-be-recognized gesture image according to an orthogonal triangular decomposition algorithm and a tuck tensor decomposition algorithm to obtain a gesture feature set of the to-be-recognized gesture image. And inputting the gesture feature set of the gesture image to be recognized into the gesture recognition model to obtain a gesture recognition result. According to the method, the cyclic convolutional neural network can be improved, and the information of each signal mode is projected to a smaller subspace by using the full convolutional neural network, so that the method has higher accuracy in gesture recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a gesture recognition method, a gesture recognition system, a computer device and a readable storage medium. BACKGROUND

[0002] At present, there are many gesture recognition methods, and the main method is to extract features of color or texture information of gestures of RGB images, and use a convolutional neural network to classify gestures. However, when analyzing color and texture information of a single gesture, the overall information of the gesture data collected multiple times is easily lost, especially when the features are extracted, distortion is prone to occur, and the convolutional neural network is prone to ignore the structural information of the gesture, which inevitably leads to a decrease in the accuracy of gesture classification. This gesture recognition analysis method analyzes the color and texture information of multiple gestures based on RGB images one by one, without considering these signals as a whole, which inevitably leads to a loss of structural information of the data, and thus a feature distortion occurs.

[0003] In the application of recognizing and classifying RGB gesture sequences, the reusability and robustness of the model face two major challenges. On the one hand, the limb reactions of different users have a greater impact on the experiment, and the gesture data collected multiple times by the same user also has great differences. On the other hand, it is necessary to calibrate the gesture data of the user for a long time.

[0004] In addition, in the recurrent convolutional neural network, although the network combines the advantages of convolutional neural networks and recurrent neural networks, the convolutional module does not process the structural information in the image sequence, and there is a certain defect in feature optimization. SUMMARY

[0005] The technical problem to be solved by the present application is that the gesture recognition analysis method in the related art may have a feature distortion, and in the application of recognizing and classifying RGB gesture sequences, the reusability and robustness of the model are poor, and the recurrent convolutional neural network has a certain defect in feature optimization.

[0006] In view of the above deficiencies of the prior art, the following solutions are provided:

[0007] In a first aspect, the present application provides a gesture recognition method, comprising: obtaining a gesture image to be recognized. Preprocessing the gesture image to be recognized according to a target detection algorithm, a single Gaussian model and a median filtering algorithm to obtain a preprocessed gesture image to be recognized. Extracting features of the preprocessed gesture image to be recognized according to an orthogonal triangular decomposition algorithm and a Tucker tensor decomposition algorithm to obtain a gesture feature set of the gesture image to be recognized. And inputting the gesture feature set of the gesture image to be recognized into a gesture recognition model to obtain a gesture recognition result. Wherein the gesture image to be recognized includes multiple frames of red, green and blue (RGB) images. The gesture recognition model is obtained by using a transfer learning method, a fully convolutional neural network and a bidirectional long short-term memory network according to a gesture image data set.

[0008] Optionally, preprocessing the gesture image to be recognized according to the target detection algorithm, the single Gaussian model and the median filtering algorithm to obtain the preprocessed gesture image to be recognized, comprising: tracking the gesture in the gesture image to be recognized using the target detection algorithm, performing skin color segmentation on image information in a spatial position of the gesture in the gesture image to be recognized and a boundary box using the single Gaussian model, and eliminating image information other than gesture edge and contour information using the median filtering algorithm to obtain the gesture edge and contour information in the gesture image to be recognized. The preprocessed gesture image to be recognized includes the gesture edge and contour information.

[0009] Optionally, extracting features of the preprocessed gesture image to be recognized according to the orthogonal triangular decomposition algorithm and the Tucker tensor decomposition algorithm to obtain the gesture feature set of the gesture image to be recognized, comprising: obtaining third-order tensor data according to an input tensor. Generating three initial factor matrices by performing orthogonal triangular decomposition on the third-order tensor data respectively. And obtaining three updated factor matrices by performing orthogonal iterative updating on the three initial factor matrices respectively. Wherein the input tensor includes the gesture edge and contour information in the gesture image to be recognized. The gesture feature set of the gesture image to be recognized includes the three updated factor matrices.

[0010] Optionally, the process of orthogonal iterative updating comprises:

[0011] iterative updating. Wherein, are the updated results of the three factor matrices in the kth iteration respectively. k is the number of iterations. X is the input tensor. ×1, ×2 and ×3 represent tensor matrixization operations in different directions, which are used to project the tensor X into the orthogonal space of each factor matrix. are the conjugate transposes of the three factor matrices in the (k-1)th iteration respectively. are orthogonal matrices in the orthogonal triangular decomposition results of the three factor matrices, respectively. are upper triangular matrices in the orthogonal triangular decomposition results of the three factor matrices, respectively. are obtained by parallel unfolding, are three updated factor matrices, respectively. denotes the first N columns of the matrix denotes the first N columns of the matrix denotes the first N columns of the matrix denotes the first N columns of the matrix denotes the first N columns of the matrix denotes the first N columns of the matrix

[0012] Optionally, the gesture recognition method further comprises: obtaining a gesture image dataset. The gesture image dataset comprises a source domain dataset and a target domain dataset. And obtaining the gesture recognition model according to the gesture image dataset by using a transfer learning method, a full convolutional neural network and a bidirectional long short-term memory network.

[0013] Optionally, obtaining the gesture recognition model according to the gesture image dataset by using the transfer learning method, the full convolutional neural network and the bidirectional long short-term memory network comprises: obtaining a source domain dataset and a target domain dataset. Preprocessing the source domain dataset and the target domain dataset according to a target detection algorithm, a single Gaussian model and a median filtering algorithm respectively to obtain a preprocessed source domain dataset and a preprocessed target domain dataset. Extracting features of the preprocessed source domain dataset and the preprocessed target domain dataset according to an orthogonal triangular decomposition algorithm and a Tucker tensor decomposition algorithm respectively to obtain a gesture feature set of the source domain dataset and a gesture feature set of the target domain dataset. Converting a covariance matrix corresponding to the gesture feature set of the source domain dataset into a covariance matrix corresponding to the gesture feature set of the target domain dataset to obtain a source domain dataset after transfer learning. And training the full convolutional neural network and the bidirectional long short-term memory network according to the source domain dataset after transfer learning to obtain the gesture recognition model. Wherein the gesture image dataset comprises the source domain dataset and the target domain dataset.

[0014] Optionally, converting the covariance matrix corresponding to the gesture feature set of the source domain dataset into the covariance matrix corresponding to the gesture feature set of the target domain dataset to obtain the source domain dataset after transfer learning comprises: obtaining the source domain dataset after transfer learning according to formula and Wherein, is a feature matrix of the i th sample in the source domain dataset after transfer learning, R TS is a mapping matrix from the gesture feature set of the target domain dataset to the gesture feature set of the source domain dataset, X Siis a feature matrix of the i-th sample in the gesture feature set of the source domain dataset, is a covariance matrix of the gesture feature set of the source domain dataset, n S is a number of samples in the gesture feature set of the source domain dataset, is a covariance matrix of the i-th gesture feature set in the gesture feature set of the source domain dataset, is is a transpose matrix of is a transpose matrix of X Si is a transpose matrix of R TS T is a transpose matrix of R TS is a transpose matrix of R. is a normalized form of a covariance matrix of the gesture feature set of the target domain dataset, is a normalized form of a covariance matrix of the gesture feature set of the source domain dataset.

[0015] Optionally, the full convolutional neural network comprises a plurality of full convolutional neural network units, each of which comprises, in sequence, a causal convolutional layer with a convolution kernel of 32@11x1, a plurality of causal expansion blocks, an activation function layer PReLU, and a plurality of multi-kernel blocks. Each causal expansion block comprises, in sequence from input to output, a dilated causal convolutional layer with a convolution kernel of 32@7x1, an activation function layer PReLU, a normalization processing layer, a dilated causal convolutional layer with a convolution kernel of 32@7x1, an activation function layer PReLU, and a normalization processing layer. Each multi-kernel block comprises, in sequence from input to output, three parallel convolution branches, a convolutional layer with a convolution kernel of 64@1x1, an activation function layer PReLU, and a normalization processing layer. The three parallel convolution branches comprise a first convolution branch, a second convolution branch, and a third convolution branch. The first convolution branch comprises, in sequence and in series, a convolutional layer with a convolution kernel of 64@1x1 and an average pooling layer with a convolution kernel of 64@2x1. The second convolution branch comprises, in sequence and in series, a convolutional layer with a convolution kernel of 64@3x1, an average pooling layer with a convolution kernel of 64@2x1, and a convolutional layer with a convolution kernel of 64@3x3. The third convolution branch comprises, in sequence and in series, a convolutional layer with a convolution kernel of 64@5x1, an average pooling layer with a convolution kernel of 64@2x1, and a convolutional layer with a convolution kernel of 64@5x5.

[0016] Optionally, the full convolutional neural network and the bidirectional long short-term memory network are trained according to the source domain data set after the transfer learning to obtain the gesture recognition model, including: using the full convolutional neural network to perform dimension reduction processing on the source domain data set after the transfer learning, and extracting the causal relationship between each feature in the source domain data set after the transfer learning to obtain a full connection feature set of the source domain data. The time feature of the full connection feature set of the source domain data is extracted using a multi-layer bidirectional long short-term memory network to obtain a time feature set of the source domain data set. The time feature set of the source domain data set is processed using a dense connection layer. And the parameters of the full convolutional neural network and the parameters of the multi-layer bidirectional long short-term memory network are optimized according to the output of the dense connection layer to obtain the gesture recognition model.

[0017] Optionally, the gesture feature set of the gesture image to be recognized is input into the gesture recognition model to obtain a gesture recognition result, including: in response to the gesture recognition model detecting a first preset number of continuous first standard gestures according to the gesture feature set of the gesture image to be recognized, determining that the first standard gesture is recognized. The first standard gesture is one gesture in a standard gesture library.

[0018] In a second aspect, the present application provides a gesture recognition system, which includes an image to be recognized acquisition module, a preprocessing module, a feature extraction module and a gesture recognition module. The image to be recognized acquisition module is configured to acquire a gesture image to be recognized. The gesture image to be recognized includes multiple red, green and blue (RGB) images. The preprocessing module is configured to preprocess the gesture image to be recognized according to a target detection algorithm, a single Gaussian model and a median filter algorithm to obtain a preprocessed gesture image to be recognized. The feature extraction module is configured to extract features of gesture edges and contour information in the gesture image to be recognized according to an orthogonal triangular decomposition algorithm and a Tucker tensor decomposition algorithm to obtain a gesture feature set of the gesture image to be recognized. The gesture recognition module is configured to input the gesture feature set of the gesture image to be recognized into a gesture recognition model to obtain a gesture recognition result. The gesture recognition model is obtained according to a gesture image data set using a transfer learning method, a full convolutional neural network and a bidirectional long short-term memory network.

[0019] In a third aspect, the present application provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the gesture recognition method.

[0020] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program. When the processor executes the computer program, the processor executes the gesture recognition method.

[0021] The gesture recognition method, system, computer device and readable storage medium provided by the application apply orthogonal triangular decomposition algorithm, Tucker tensor decomposition algorithm, transfer learning algorithm, full convolutional neural network, bidirectional long short-term memory network and the like for gesture recognition, can improve the cyclic convolutional neural network, project the information of each signal mode to a smaller subspace by using the full convolutional neural network, and make the gesture recognition have higher accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of a gesture recognition method in an embodiment of the application;

[0023] Figure 2 A flowchart of another gesture recognition method in an embodiment of the application;

[0024] Figure 3 A flowchart of a parallel triangular orthogonal iterative Tucker tensor decomposition method in a gesture recognition method in an embodiment of the application;

[0025] Figure 4 A flowchart of a traditional high-order orthogonal iterative algorithm in the prior art;

[0026] Figure 5 An iterative flowchart of a parallel triangular orthogonal iterative Tucker tensor decomposition method in a gesture recognition method in an embodiment of the application;

[0027] Figure 6 An iterative flowchart of a traditional high-order orthogonal iterative algorithm in the prior art;

[0028] Figure 7 A network structure diagram of a gesture recognition model in still another gesture recognition method in an embodiment of the application;

[0029] Figure 8 A flowchart of still another gesture recognition method in an embodiment of the application;

[0030] Figure 9 A structure diagram of a full convolutional neural network unit in a gesture recognition method in an embodiment of the application;

[0031] Figure 10 A principle schematic diagram of a causal expansion block in a gesture recognition method in an embodiment of the application;

[0032] Figure 11 A flowchart of still another gesture recognition method in an embodiment of the application;

[0033] Figure 12 A full flowchart of a gesture recognition method in an embodiment of the application;

[0034] Figure 13A schematic diagram of a full flow of another gesture recognition method in an embodiment of the present application;

[0035] Figure 14 An architecture diagram of a gesture recognition system in an embodiment of the present application;

[0036] Figure 15 An architecture diagram of another gesture recognition system in an embodiment of the present application;

[0037] Figure 16 A structural diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to enable a person skilled in the art to better understand the technical solutions of the present application, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0039] It can be understood that the specific embodiments and drawings described herein are merely used to explain the present application, but not to limit the present application.

[0040] It can be understood that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0041] It can be understood that, for the convenience of description, only the parts related to the present application are shown in the drawings of the present application, and the parts unrelated to the present application are not shown in the drawings.

[0042] It can be understood that each unit and module involved in the embodiments of the present application can correspond to only one entity structure, or can be composed of multiple entity structures, or multiple units and modules can be integrated into one entity structure.

[0043] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present application can occur in an order different from that marked in the drawings.

[0044] It can be understood that, in the flowcharts and block diagrams of the present application, the system, device, equipment, method according to the embodiments of the present application are shown as possible implementation architecture, function and operation. Each block in the flowchart or block diagram can represent a unit, module, program segment, code, which contains executable instructions for realizing the specified function. Moreover, each block or combination of blocks in the block diagram and flowchart can be realized by a hardware-based system for realizing the specified function, or by a combination of hardware and computer instructions.

[0045] It can be understood that the units and modules involved in the embodiments of the present application can be realized in the form of software or hardware, for example, the units and modules can be located in a processor.

[0046] Some embodiments of the present application provide a gesture recognition method, such as Figure 1 As shown, the method comprises steps 101 to 104.

[0047] Step 101, obtaining a gesture image to be recognized.

[0048] In step 101, the gesture image to be recognized comprises a plurality of red-green-blue (RGB) images.

[0049] It can be understood that the gesture image to be recognized can be pre-acquired and saved, or acquired in real time.

[0050] Illustratively, the gesture image to be recognized can be a video image, and the video image comprises a plurality of red-green-blue (RGB) images.

[0051] It can be understood that an RGB image is a digital image composed of red (R), green (G), and blue (B) color channels. It is the most commonly used and basic color image representation method, and is widely used in display, digital camera, web design, etc.

[0052] Step 102, pre-processing the gesture image to be recognized according to a target detection algorithm, a single Gaussian model, and a median filter algorithm to obtain a pre-processed gesture image to be recognized.

[0053] It can be understood that the target detection algorithm aims to identify the category of a specific object in an image and determine its location (usually represented by a rectangular bounding box). Illustratively, the target detection algorithm used in some embodiments of the present application is the YOLO (You Only Look Once) algorithm.

[0054] The YOLO algorithm regards target detection as a single regression problem and directly operates on the entire input image. It uniformly divides the image into an S×S grid, and each grid cell is responsible for predicting a fixed number of bounding boxes (BB), as well as the confidence score and class probability of the objects contained in these boxes. The YOLO algorithm simultaneously predicts the location and class of objects that may exist at all positions in the image in one forward propagation process, achieving "one look" detection.

[0055] Single Gaussian Model (SGM) is a probabilistic generative model, whose core assumption is that all observations follow the same multivariate Gaussian distribution. The model learns the two parameters from training samples by maximum likelihood estimation, and then calculates the probability of new samples belonging to the distribution. SGM is often used in anomaly detection tasks because data with significantly lower probability than the core region of the distribution is judged as abnormal. It can also be used as a basic component of generative classification models (e.g., modeling each class independently).

[0056] The median filtering algorithm is a nonlinear image denoising method, whose core operation is based on the order statistics theory. The median filtering algorithm scans the image pixel by pixel, extracts a fixed-size neighborhood window (such as 3x3 or 5x5) centered on the current pixel at each position, arranges all pixel values in the window in ascending order of intensity, and selects the median value in the sorted order to replace the original value of the center pixel. This process traverses the entire image through a sliding window to achieve noise filtering.

[0057] Exemplarily, the YOLO algorithm can be used to first locate the hand boundary box, then the single Gaussian skin color model can be used to quickly segment the finger region in the box located by the YOLO algorithm, and finally the median filtering algorithm can be used for denoising.

[0058] In some embodiments, the implementation method of step 102 includes: tracking the hand gesture in the hand gesture image to be recognized using a target detection algorithm, performing skin color segmentation on the image information in the spatial position and the boundary box of the hand gesture in the hand gesture image to be recognized using a single Gaussian model, and eliminating the image information other than the edge and contour information of the hand gesture using a median filtering algorithm to obtain the edge and contour information of the hand gesture in the hand gesture image to be recognized. The preprocessed hand gesture image to be recognized includes the edge and contour information of the hand gesture.

[0059] Step 103, performing feature extraction on the edge and contour information of the hand gesture in the hand gesture image to be recognized according to the orthogonal triangular decomposition algorithm and the Tucker tensor decomposition algorithm to obtain a hand gesture feature set of the hand gesture image to be recognized.

[0060] It can be understood that the orthogonal triangular decomposition (also known as QR decomposition, English full name: Quorthogonal-Right Triangular Decomposition, English abbreviation: QR Decomposition or QRD) is an algorithm for decomposing any real or complex matrix A into the product of two specific matrices: A = QR. Where Q is an orthogonal matrix (satisfying Q TQ = 1, the column vectors are orthogonal and have a modulus of 1), and R is an upper triangular matrix (all elements below the main diagonal are zero). The core advantage of the QR decomposition is its numerical stability, making it a cornerstone for solving linear equations, least squares problems, and computing eigenvalues ​​(e.g., the QR iterative algorithm).

[0061] As you can understand, Tucker tensor decomposition is a generalized low-rank approximation method for high-order tensors (multidimensional arrays). It decomposes the original tensor into a core tensor and a set of factor matrices, achieving tensor dimensionality reduction and feature extraction through adjustable rank.

[0062] In some embodiments, as Figure 2 、 Figure 3 As shown in Table 1, the implementation method of step 103 includes steps 201 to 203.

[0063] Step 201: Obtain third-order tensor data according to the input tensor.

[0064] In step 201 , the input tensor includes gesture edge and contour information in the gesture image to be recognized.

[0065] It can be understood that the gesture edge and contour information in the gesture image is obtained based on the gesture image to be recognized. Therefore, the gesture edge and contour information in the gesture image is a three-dimensional information. This information is used as the input tensor and expanded using three modes respectively to obtain third-order tensor data.

[0066] Step 202: Perform orthogonal triangular decomposition on the third-order tensor data to generate three initial factor matrices.

[0067] The implementation method of step 202 is shown in equations (1) to (6).

[0068] Q (1) R (1) =X (1) (1)

[0069]

[0070] In formula (1) to formula (6), X (1) 、X (2) 、X (3) They are Figure 3 The third-order tensor data obtained by expanding the input tensor in three modes, Q (1) , Q (2) , Q (3) They are the orthogonal matrices obtained from the first iteration of the third-order tensor data, R (1) 、R (2) 、R (3) They are the upper triangular matrices obtained from the first iteration of the third-order tensor data. are three initial factor matrices, respectively.

[0071] Step 203, orthogonally iteratively updating the three initial factor matrices to obtain three updated factor matrices.

[0072] In step 203, the gesture feature set of the gesture image to be recognized includes the three updated factor matrices.

[0073] In some embodiments, in step 203, the process of orthogonally iteratively updating includes iteratively updating using formulas (7) to (15).

[0074]

[0075]

[0076] In formulas (7) to (15), are updated results of the three factor matrices in the kth iteration. k is the iteration number, and X is the input tensor. ×1, ×2, and ×3 represent tensor matrix operations in different directions, which are used to project the tensor X into the orthogonal space of each factor matrix. are conjugate transposes of the three factor matrices in the (k-1)th iteration. are orthogonal matrices in the orthogonal triangular decomposition results of the three factor matrices. are upper triangular matrices in the orthogonal triangular decomposition results of the three factor matrices. are matrices obtained by parallel unfolding, are three updated factor matrices. represents extracting the first N columns from the matrix represents extracting the first N columns from the matrix represents extracting the first N columns from the matrix

[0077] From k=1 to k=K, iteratively according to formulas (7) to (15) to obtain the core tensor

[0078]

[0079] Exemplarily, as Figure 3 ​​​​As shown in Table 1, the gesture recognition method proposed in some embodiments of the present invention uses a parallel triangular orthogonal iterative Tucker tensor decomposition method to implement step 103. This method can also be called a parallel QR orthogonal iterative (PQROI) Tucker tensor decomposition method or a parallel QR orthogonal iterative algorithm.

[0080] Table 2 Parallel QR orthogonal iterative algorithm

[0081]

[0082]

[0083] like Figure 3 As shown in Table 1, first, the input tensor is expanded in three modes to obtain X (1) , X (2) and X (3) Afterwards, QR decomposition is performed on each unfolded matrix to generate three factor matrices and In addition, if Figure 3 As shown, the tensor and In each iteration, the matrix and Tensors calculated simultaneously. and Parallel expansion into matrices and Finally, based on and Apply QRD to simultaneously obtain the updated factor matrix and

[0084] It is understandable that the conventional high-order orthogonal iteration (HOOI) algorithm in the prior art is as follows: Figure 4 As shown in Table 2.

[0085] Table 2 Traditional high-order orthogonal iterative algorithm

[0086]

[0087]

[0088] According to Table 2, equations (17) to (32) can be obtained.

[0089] Q (1) R (1) =X (1)(17)

[0090]

[0091]

[0092] From k=1 to k=K, iterate according to equations (23) to (31), and the obtained core tensor is shown in equation (32).

[0093]

[0094] according to Figure 5 From (17) to (32), the matrix U updated in the traditional HOOI algorithm (23) at the kth iteration is (1) Recorded as The updated matrix U at the (k-1)th iteration (3) Recorded as That is, in each iteration, we must first obtain U (1) , and then we can calculate the tensor Similarly, as shown in Equation (26) of the traditional HOOI algorithm, the updated matrix U at the kth iteration is (1) and U (2) Recorded as and That is, when calculating the tensor Before, you must complete and It can be seen that the update sequence of the factor matrix in the traditional HOOI algorithm is as follows Figure 5 As shown in Figure 2, in the HOOI algorithm, the factor matrix is ​​updated sequentially, which will lead to data dependency problems.

[0095] And as Figure 3 、 Figure 6 As shown in equations (1) to (16), in some embodiments of the present invention, in the orthogonal iterative update process, in equation (10) at the kth iteration, the tensor Is to use

[0096] (k-1) iterations of U (1) (Right now ) instead of using the k-th repetition In short, tensors The generation of at the kth iteration does not depend on Furthermore, the equations (7) to (9) shown in The update of and formula (10) to formula (11) shown in The updates of are independent of each other, that is, and The update operations of and can be performed simultaneously. In addition, as shown in equation (13), the tensor is calculated by using and It can be seen that the update of is independent of the update of or The update of the factor matrix can be performed in parallel. That is, the update of the factor matrix can be performed simultaneously and independently of each other. Obviously, some embodiments of the present application use a low-latency tensor decomposition algorithm, which can solve the data dependency problem in the traditional HOOI algorithm.

[0097] Step 104, input the gesture feature set of the gesture image to be recognized into the gesture recognition model to obtain a gesture recognition result.

[0098] In step 104, the gesture recognition model is obtained according to the gesture image data set by using a transfer learning method, a full convolutional neural network, and a bidirectional long short-term memory network.

[0099] It can be understood that the transfer learning method is a machine learning paradigm that uses the learned knowledge of a source domain to improve the performance of a target domain model. The core is to break through the assumption of independent and identically distributed data of traditional models, and solve the challenge of data scarcity or insufficient annotation in the target task by transferring the feature representation, model parameters or domain invariant features in the source task. The transfer learning method can significantly reduce the dependence of the target domain on large-scale labeled data and improve the generalization ability in small sample scenarios.

[0100] The core idea of the full convolutional neural network (FCN) is to replace all the fully connected layers at the end of the traditional convolutional neural network (CNN) with convolutional layers. In this way, the FCN can accept input images of any size and output a two-dimensional feature map (such as a segmentation map, a heat map, etc.) that retains the spatial dimension, rather than a fixed-length feature vector output by traditional CNN. FCN usually uses transposed convolution (deconvolution) operation to upsample deep and abstract low-resolution feature maps, and combines high-resolution feature information from the shallow layer of the network (through skip connection), and finally generates a fine-grained pixel-level prediction result that matches the size of the input image.

[0101] Bidirectional Long Short-Term Memory (BiLSTM) is an improved Recurrent Neural Network (RNN), the core innovation of which is to connect two independent LSTM layers in series, which are the forward layer and the reverse layer. The forward layer processes the sequence in time order to capture historical dependencies, and the reverse layer processes the sequence in reverse order to extract future context information. The final output of each time step is generated by concatenation or weighted fusion of the bidirectional layer states. This structure alleviates gradient vanishing through the gating mechanism (input gate / forget gate / output gate), while using bidirectional information flow to enhance the modeling ability of long-distance dependencies.

[0102] In some embodiments, the gesture recognition method further comprises: obtaining a gesture image dataset, and obtaining the gesture recognition model by using a transfer learning method, a fully convolutional neural network, and a bidirectional long short-term memory network according to the gesture image dataset. The gesture image dataset includes a source domain dataset and a target domain dataset.

[0103] Exemplarily, as shown in Figure 7 The network structure of the gesture recognition model is composed of four parts: a transfer learning module, a fully convolutional neural network, a bidirectional LSTM network, and a dense connection layer. First, the improved Euclidean alignment (MEA) operation is used to minimize the difference of the mapped subject data on the "average covariance matrix" (for details, refer to equations (33) to (39)), then the FCN is used for feature dimension reduction and causal relationship extraction, then the multi-layer bidirectional long short-term memory network extracts the time features that should change in gesture recognition, and finally the dense connection layer is used for output.

[0104] In some embodiments, as shown in Figure 8 The implementation of the gesture recognition model by using the transfer learning method, the fully convolutional neural network, and the bidirectional long short-term memory network according to the gesture image dataset includes steps 801 to 806.

[0105] Step 801, obtaining a source domain dataset and a target domain dataset.

[0106] In step 801, the gesture image dataset includes a source domain dataset and a target domain dataset.

[0107] It can be understood that, as Figure 12As shown, a plurality of gesture image data of a plurality of subjects can be acquired first, and a gesture image dataset can be obtained according to the plurality of gesture image data of the plurality of subjects. Then, a part (for example, 10%) of the data in the gesture image dataset is randomly selected as test data, and a test dataset is formed. The remaining part (for example, 90%) of the data in the gesture image dataset is determined as target domain data (for example, by means of a label or the like), and a target domain dataset is formed. Finally, the other data in the gesture image dataset can be used as source domain data, and a source domain dataset is formed.

[0108] In step 802, the source domain dataset and the target domain dataset are preprocessed according to a target detection algorithm, a single Gaussian model and a median filter algorithm respectively, to obtain a preprocessed source domain dataset and a preprocessed target domain dataset.

[0109] It can be understood that the implementation method of step 802 can refer to the related description of step 102, which will not be repeated here.

[0110] In step 803, feature extraction is performed on the preprocessed source domain dataset and the preprocessed target domain dataset according to a orthogonal triangular decomposition algorithm and a Tucker tensor decomposition algorithm respectively, to obtain a gesture feature set of the source domain dataset and a gesture feature set of the target domain dataset.

[0111] It can be understood that the implementation method of step 803 can refer to the related description of step 103, which will not be repeated here.

[0112] In step 804, the average covariance matrix corresponding to the gesture feature set of the source domain dataset is converted into an average covariance matrix corresponding to the gesture feature set of the target domain dataset, to obtain a source domain dataset after transfer learning.

[0113] In some embodiments, the implementation method of step 804 can be: obtaining the source domain feature set after transfer learning according to formula (33) and formula (34).

[0114]

[0115] Formula (33) and formula (34), is the feature matrix of the i-th sample in the source domain dataset after transfer learning, R TS is a mapping matrix from the gesture feature set of the target domain dataset to the gesture feature set of the source domain dataset, X Si is the feature matrix of the i-th sample in the gesture feature set of the source domain dataset, i is the sample serial number in the gesture feature set of the source domain dataset, is the covariance matrix of the gesture feature set of the source domain dataset, n S is the number of samples in the gesture feature set of the source domain dataset, is the covariance matrix of the i-th gesture feature set in the gesture feature set of the source domain dataset, is the transpose matrix of is the transpose matrix of Si TS T is the transpose matrix of TS is the normalized form of the covariance matrix of the gesture feature set of the target domain dataset, is the normalized form of the covariance matrix of the gesture feature set of the source domain dataset.

[0116] It can be understood that according to formula (34), the average covariance matrix corresponding to the previous source domain subject is transformed into the average covariance matrix corresponding to the target domain subject by using the improved Euclidean alignment. After the improved Euclidean alignment operation is performed on the data, the difference of the mapped subject data on the "average covariance matrix" is minimized, so as to further implicitly reduce the difference of the original distribution.

[0117] It can be understood that, taking the target domain dataset including n T image data as an example, the n T gesture feature sets of the target domain dataset are For example, X Tj is the j-th gesture feature in the n T gesture features of the target domain dataset.

[0118] First, the covariance matrix of each gesture feature in the n T gesture features in the n T gesture features of the target domain dataset is calculated according to formula (35).

[0119]

[0120] In formula (35), is the transpose matrix of Tj X Tj is the covariance matrix of Tj X

[0121] Subsequently, the average covariance matrix T of the n gesture features in the n gesture features of the target domain dataset is calculated according to formula (36).

[0122]

[0123] Meanwhile, taking the source domain dataset including n S image data as an example, the gesture feature set of the source domain dataset is For example, X Sj ​is the i-th gesture feature of the n S gesture features of the source domain dataset.

[0124] First, the covariance matrix of each feature in the i-th gesture feature of the n S gesture features of the experimental subject in the source domain dataset is calculated according to formula (37).

[0125]

[0126] In formula (35), is the transpose matrix of X Si , C Si is the covariance matrix of X Si .

[0127] Subsequently, the average covariance matrix of the n S gesture features of the source domain dataset is calculated according to formula (38).

[0128]

[0129] Then, the mapping matrix R TS from the gesture feature set of the target domain dataset to the gesture feature set of the source domain dataset is calculated, as shown in formula (39).

[0130]

[0131] Then, the Euclidean alignment is performed through formula (33), and the average covariance matrix corresponding to the previous source domain subject is converted into the average covariance matrix corresponding to the target domain subject, as shown in formula (35).

[0132] It can be understood that, after the improved Euclidean alignment operation is performed on the data, the difference of the mapped subject data on the "average covariance matrix" is also minimized, so that the difference of the original distribution is further implicitly reduced. In addition, this method only needs to perform alignment on the data of the source domain subject, which is more conducive to gesture recognition.

[0133] Step 805: training the full convolutional neural network and the bidirectional long short-term memory network according to the source domain dataset after the transfer learning, to obtain a gesture recognition model.

[0134] In some embodiments, as Figure 9As shown, the fully convolutional neural network includes multiple fully convolutional neural network units. Each fully convolutional neural network unit sequentially includes a causal convolution layer with a convolution kernel of 32@11x1, multiple causal expansion blocks, a Prelude (PReLU) activation function layer, and multiple multi-core blocks. Each causal expansion block, from input to output, sequentially includes a dilated causal convolution layer with a convolution kernel of 32@7x1, a Prelude (PReLU) activation function layer, a normalization layer, a dilated causal convolution layer with a convolution kernel of 32@7x1, a Prelude (PReLU) activation function layer, and a normalization layer. Each multi-core block, from input to output, sequentially includes three parallel convolution branches: a convolution layer with a convolution kernel of 64@1x1, a Prelude (PReLU) activation function layer, and a normalization layer. The three parallel convolution branches include the first convolution branch, the second convolution branch, and the third convolution branch. The first convolution branch includes a convolution layer with a convolution kernel of 64@1x1 and an average pooling layer with a convolution kernel of 64@2x1, sequentially. The second convolution branch consists of a convolution layer with a kernel size of 64@3x1, an average pooling layer with a kernel size of 64@2x1, and a convolution layer with a kernel size of 64@3x3. The third convolution branch consists of a convolution layer with a kernel size of 64@5x1, an average pooling layer with a kernel size of 64@2x1, and a convolution layer with a kernel size of 64@5x5.

[0135] It can be understood that the number of causal dilation blocks (CDB) and the number of multi kernel blocks (MKB) in each fully convolutional neural network unit can be the same or different.

[0136] Understandably, at any moment of gesture recognition, all previous gesture history must be considered to distinguish between the changes in motion that trigger different gesture events. Furthermore, to correctly identify different gesture events, the changes in the RGB image sequence should be analyzed over a longer time window. Because short-term fluctuations at different stages of a gesture can lead to incorrect recognition, the RGB image sequence should be analyzed from different observation ranges.

[0137] like Figure 10 As shown in Figure 2, CDB achieves the first goal by utilizing a series of dilated causal convolutions, because the causal convolution operation at any time takes into account all previous history. However, using only causal convolutions would require a very deep architecture to achieve a large receptive field. Therefore, by utilizing dilated convolutions, the receptive field can be significantly increased.

[0138] If the input signal corresponds to x∈R nAnd the filter of length T is denoted as f:{0, 1,..., T-1}→R, then the causal dilated convolution operation F on any element s is given by equation (40):

[0139]

[0140] In equation (40), * denotes the causal dilated convolution with dilation rate d, and (s-d) denotes the causal convolution with previous time stamp. Therefore, the effective receptive field in one such layer is (T-1)dt block, and the dilation rate varies exponentially with (d=O(2 d )). t )) in each block.

[0141] In each such block, two causal dilated convolutions are concatenated, a parametric rectified linear unit (PRelu) is applied as a nonlinear activation function, and then a normalization operation is performed, as shown in equation (41).

[0142]

[0143] In equation (41), a is the slope used to map negative input values. For example, a=0.2 can make the convergence speed faster. If f l is the output feature map obtained from the lth CDB, a residual output R l will also be generated, which will be fed into the CDB as shown in equation (42).

[0144]

[0145] In equation (42), R0=x l is the input signal, and the initial value is x1, when x1=1, R l -1=R0, and the next value is R1=f1+R0, and L is the total number of causal dilated blocks. Then, the features extracted by each CDB unit block are combined to generate the combined output F, as shown in equation (43)

[0146]

[0147] After processing the multiple causal expansion blocks, the input data corresponding to the RGB image tensor features is converted into a composite feature map containing extracted features of multiple different temporal relationships. However, the temporal dimension of this feature map remains the same as the input data frame, which should be further reduced to extract more general temporal features. Therefore, the MKB is proposed to reduce the temporal dimension of the feature map while performing multi-kernel convolution operations in parallel. Here, the average pooling operation is performed in parallel between convolutions with different kernels, which helps to merge different temporal contexts in the pooling operation. Subsequently, all these pooled and convolved feature maps converge with another convolution operation that extracts general variations. Therefore, the output of each MKB unit can be summarized as shown in equation (44).

[0148]

[0149] where O0= F; h1, h2, h3 represent convolutions with temporal kernels of size 1, 3, and 5, respectively; H represents a combined convolution, θ1, θ2, θ3, and θ H are their respective parameters, and M is the total number of such MKB units. After passing through a series of multi-kernel blocks, the transformed feature map will contain more and more generalized features, and its temporal dimension will be reduced.

[0150] In some embodiments, as shown in FIG. 11, the implementation method of step 805 includes steps 1101-1104. Figure 11

[0151] Step 1101, using a fully convolutional neural network to reduce dimensionality of the source domain dataset after transfer learning and extract causal relationships between features in the source domain dataset after transfer learning, to obtain a fully connected feature set of the source domain data.

[0152] It can be understood that the fully convolutional neural network can reduce dimensionality of the source domain dataset after transfer learning and extract causal relationships between features in the source domain dataset after transfer learning, and the specific processing process can refer to the related description of step 805, which will not be repeated here.

[0153] Step 1102, using a multi-layer bidirectional long short-term memory network to extract temporal features of the fully connected feature set of the source domain data, to obtain a temporal feature set of the source domain dataset.

[0154] It can be understood that the multi-layer bidirectional long short-term memory network can extract temporal features of gesture changes from the feature maps obtained from different FCN units to obtain a temporal feature set of the source domain dataset.

[0155] Step 1103, using a densely connected layer to process the temporal feature set of the source domain dataset.

[0156] ​It can be understood that the dense connection layer is placed before the output layer, which can map the integrated feature maps to the input dimension required by the output layer, and "prepare" the input with correct format and relevant information for the output layer.

[0157] Step 1104, optimizing the parameters of the full convolutional neural network and the parameters of the multi-layer bidirectional long short-term memory network according to the output of the dense connection layer, to obtain a gesture recognition model.

[0158] It can be understood that the training is actually a process of optimizing the parameters of each network structure, and after a large amount of data set is trained, the gesture recognition model can be obtained.

[0159] In some embodiments, the implementation method of step 104 includes: determining that the first standard gesture is recognized in response to the gesture recognition model detecting a first preset number of continuous first standard gestures from the gesture feature set of the gesture image to be recognized. The first standard gesture is one gesture in the standard gesture library.

[0160] It can be understood that the standard gesture library can save a plurality of standard gestures, and by comparing the output of the gesture recognition model with the standard gestures, it can be determined whether the output of the gesture recognition model belongs to the standard gestures and which standard gestures it belongs to.

[0161] Exemplarily, as shown in Figure 12 , the trained gesture recognition model is used to continuously recognize the test data set, and the gesture labels corresponding to 1 to 10 are respectively designated as numbers 1 to 10. The initial recognition number is set to 0. When the gesture corresponding to the number is detected, the recognition number is increased by 1, otherwise the recognition number is set to zero. For each RGB image data, if 5 continuous same gestures can be detected, that is, the recognition number is equal to 5, a trigger signal will be sent. The gesture recognition method provided by the embodiment of the application is respectively applied to the SKIG (Sheffield Kinect Gesture Dataset, Sheffield Kinect Gesture Dataset) data set and the actual use scene, and the gesture recognition accuracy in the SKIG data set reaches 98.81%. At the same time, the algorithm can realize the function of human-computer interaction in the actual environment, so the algorithm can be widely applied to the fields of human-computer interaction, virtual reality, smart home, medical health and the like.

[0162] In general, as Figure 13As shown, some embodiments of the present application provide a gesture recognition method including five stages: gesture data preprocessing, establishing an RGB image sequence tensor, tensor decomposition, fixed factor matrix feature extraction, transfer learning, and full connection-bidirectional long short-term memory network for gesture recognition classification. Some embodiments of the present application provide a gesture recognition method, the main process of which includes: first, 10% of the gesture data of each user is selected as a test set, and the rest is used as a training set. The YOLO algorithm is used to track the user's gestures and pre-process the RGB data. A single Gaussian model is selected to segment the skin color of each frame of the RGB gesture sequence. The median filtering method is used to eliminate the gesture edge and contour information while protecting the edge and contour of the gesture. Subsequently, the RGB data with a continuous time length of 1s in the database is processed to construct a three-order tensor, and the gesture features are extracted using a parallel QR orthogonal iteration algorithm. Finally, the features are put into a full convolution-bidirectional long short-term memory network for training after transfer learning. The parallel QR orthogonal iteration algorithm and the high-order orthogonal iteration algorithm are shown in Table 1 and Table 2, respectively.

[0163] In summary, the gesture recognition method provided by the embodiments of the present application improves the traditional high-order orthogonal iteration algorithm, efficiently extracts the structural information in the RGB image using the PQROI algorithm, and avoids feature distortion. At the same time, the parallel tensor decomposition method can extract tensor features with low delay and improve the feature extraction efficiency. In addition, the gesture recognition method provided by the embodiments of the present application improves the Euclidean alignment, introduces transfer learning into the gesture classification experiment, and improves the reusability of the gesture recognition model. At the same time, the recurrent convolutional neural network is improved, and the full convolutional neural network is used to project the information of each signal mode to a smaller subspace, so that it has a higher accuracy in gesture recognition.

[0164] Some embodiments of the present application provide a gesture recognition system, as shown in Figure 14 The gesture recognition system 1400 includes a to-be-recognized image acquisition module 1401, a preprocessing module 1402, a feature extraction module 1403, and a gesture recognition module 1404.

[0165] The to-be-recognized image acquisition module 1401 is configured to acquire a to-be-recognized gesture image. The to-be-recognized gesture image includes multiple frames of red-green-blue (RGB) images.

[0166] The preprocessing module 1402 is configured to pre-process the to-be-recognized gesture image according to a target detection algorithm, a single Gaussian model, and a median filtering algorithm to obtain a pre-processed to-be-recognized gesture image.

[0167] In some embodiments, the preprocessing module 1402 is configured to: track the gesture in the gesture image to be recognized using an object detection algorithm, perform skin color segmentation on the spatial location of the gesture and image information within the bounding box in the gesture image to be recognized using a single Gaussian model, and eliminate image information other than gesture edge and contour information using a median filter algorithm to obtain gesture edge and contour information in the gesture image to be recognized. The preprocessed gesture image to be recognized includes gesture edge and contour information.

[0168] The feature extraction module 1403 is configured to extract features from gesture edge and contour information in the gesture image to be recognized according to the orthogonal triangulation decomposition algorithm and the Tucker tensor decomposition algorithm to obtain a gesture feature set of the gesture image to be recognized.

[0169] In some embodiments, the feature extraction module 1403 is configured to: obtain third-order tensor data based on an input tensor; perform orthogonal triangular decomposition on the third-order tensor data to generate three initial factor matrices; and perform orthogonal iterative updates on the three initial factor matrices to obtain three updated factor matrices. The input tensor includes gesture edge and contour information in a gesture image to be recognized. The gesture feature set of the gesture image to be recognized includes the three updated factor matrices.

[0170] In some embodiments, the orthogonal iterative update process includes: using the formula Perform iterative updates. are the updated results of the three factor matrices in iteration k. k is the number of iterations. X is the input tensor. ×1, ×2, and ×3 represent matrix operations in different directions, respectively, used to project tensor X into the orthogonal space of the factor matrices. are the conjugate transposes of the three factor matrices in the k-1th iteration respectively. are the orthogonal matrices in the orthogonal triangular decomposition results of the three factor matrices. are the upper triangular matrices in the orthogonal triangular decomposition results of the three factor matrices. They are respectively The matrix obtained by parallel expansion is, They are the three updated factor matrices respectively. Represents the matrix Extract the first N columns, Represents the matrix Extract the first N columns, Represents the matrix Extract the first N columns.

[0171] In some embodiments, asFigure 15 As shown, gesture recognition system 1400 further includes a gesture recognition model acquisition module 1405. This module is configured to acquire a gesture image dataset, including a source domain dataset and a target domain dataset. Furthermore, based on the gesture image dataset, a gesture recognition model is derived using a transfer learning method, a fully convolutional neural network, and a bidirectional long short-term memory network.

[0172] In some embodiments, the gesture recognition model acquisition module 1405 is configured to: obtain a source domain dataset and a target domain dataset. Preprocess the source domain dataset and the target domain dataset using an object detection algorithm, a single Gaussian model, and a median filter algorithm, respectively, to obtain a preprocessed source domain dataset and a preprocessed target domain dataset. Feature extraction is performed on the preprocessed source domain dataset and the preprocessed target domain dataset using an orthogonal triangular decomposition algorithm and a Tucker tensor decomposition algorithm, respectively, to obtain a gesture feature set of the source domain dataset and a gesture feature set of the target domain dataset. The covariance matrix corresponding to the gesture feature set of the source domain dataset is converted into a covariance matrix corresponding to the gesture feature set of the target domain dataset, to obtain a source domain dataset after transfer learning. Furthermore, a fully convolutional neural network and a bidirectional long short-term memory network are trained based on the source domain dataset after transfer learning to obtain a gesture recognition model. The gesture image dataset includes the source domain dataset and the target domain dataset.

[0173] In some embodiments, the covariance matrix corresponding to the gesture feature set of the source domain dataset is converted into the covariance matrix corresponding to the gesture feature set of the target domain dataset to obtain the source domain dataset after transfer learning. The gesture recognition model acquisition module 1405 is set to: According to the formula and Get the source domain dataset after transfer learning. is the feature matrix of the i-th sample in the source domain dataset after transfer learning, R TS is the mapping matrix from the gesture feature set of the target domain dataset to the gesture feature set of the source domain dataset, X Si is the feature matrix of the i-th sample in the gesture feature set of the source domain dataset, is the covariance matrix of the gesture feature set of the source domain dataset, n S is the number of samples in the gesture feature set of the source domain dataset, is the covariance matrix of the i-th gesture feature set in the gesture feature set of the source domain dataset, yes The transposed matrix of . It's X Si The transposed matrix of R TS T It is R TS The transposed matrix of . is a normalized form of a covariance matrix of a gesture feature set of the target domain data set, is a normalized form of a covariance matrix of a gesture feature set of the source domain data set.

[0174] In some embodiments, the full convolutional neural network comprises a plurality of full convolutional neural network units, each of which comprises, in sequence, a causal convolutional layer with a convolution kernel of 32@11x1, a plurality of causal expansion blocks, a PReLU activation function layer, and a plurality of multi-kernel blocks. Each causal expansion block comprises, in sequence from input to output, a dilated causal convolutional layer with a convolution kernel of 32@7x1, a PReLU activation function layer, a normalization processing layer, a dilated causal convolutional layer with a convolution kernel of 32@7x1, a PReLU activation function layer, and a normalization processing layer. Each multi-kernel block comprises, in sequence from input to output, three parallel convolutional branches, a convolutional layer with a convolution kernel of 64@1x1, a PReLU activation function layer, and a normalization processing layer. The three parallel convolutional branches comprise a first convolutional branch, a second convolutional branch, and a third convolutional branch. The first convolutional branch comprises, in sequence, a convolutional layer with a convolution kernel of 64@1x1 and an average pooling layer with a convolution kernel of 64@2x1. The second convolutional branch comprises, in sequence, a convolutional layer with a convolution kernel of 64@3x1, an average pooling layer with a convolution kernel of 64@2x1, and a convolutional layer with a convolution kernel of 64@3x3. The third convolutional branch comprises, in sequence, a convolutional layer with a convolution kernel of 64@5x1, an average pooling layer with a convolution kernel of 64@2x1, and a convolutional layer with a convolution kernel of 64@5x5.

[0175] In some embodiments, in the case where the gesture recognition model is obtained by training the full convolutional neural network and the bidirectional long short-term memory network according to the source domain data set after transfer learning, the gesture recognition model obtaining module 1405 is configured to: perform dimensionality reduction processing on the source domain data set after transfer learning using the full convolutional neural network, and extract the causal relationship between each feature in the source domain data set after transfer learning to obtain a full connection feature set of the source domain data. Extract the time feature of the full connection feature set of the source domain data using the multi-layer bidirectional long short-term memory network to obtain a time feature set of the source domain data set. Process the time feature set of the source domain data set using the dense connection layer. And, optimize the parameters of the full convolutional neural network and the parameters of the multi-layer bidirectional long short-term memory network according to the output of the dense connection layer to obtain the gesture recognition model.

[0176] The gesture recognition module 1404 is configured to input the gesture feature set of the gesture image to be recognized into the gesture recognition model to obtain a gesture recognition result. The gesture recognition model is obtained by using the transfer learning method, the full convolutional neural network, and the bidirectional long short-term memory network according to the gesture image data set.

[0177] In some embodiments, the gesture recognition module 1404 is configured to determine that the first standard gesture is recognized in response to the gesture recognition model detecting a first preset number of continuous first standard gestures from the gesture feature set of the gesture image to be recognized. The first standard gesture is one gesture in a standard gesture library.

[0178] The specific scheme and beneficial effects of the gesture recognition system provided by the embodiments of the present application can be referred to the related description of the gesture recognition method provided by the embodiments of the present application, which will not be repeated here.

[0179] Some embodiments of the present application provide a computer device, such as Figure 15 As shown in the figure, the computer device 1600 includes a memory 1601 and a processor 1602, and the memory 1601 stores a computer program. When the processor 1602 runs the computer program stored in the memory 1601, the processor 1602 executes the gesture recognition method described above.

[0180] The specific scheme and beneficial effects of the computer device provided by some embodiments of the present application can be referred to the related description of the gesture recognition cloud platform provided by some embodiments of the present application, which will not be repeated here.

[0181] Some embodiments of the present application provide a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the gesture recognition method described above.

[0182] The specific scheme and beneficial effects of the computer readable storage medium provided by some embodiments of the present application can be referred to the related description of the gesture recognition cloud platform provided by some embodiments of the present application, which will not be repeated here.

[0183] It can be understood that the above embodiments are only exemplary embodiments adopted for illustrating the principles of the present application, and the present application is not limited thereto. Various modifications and improvements can be made by those skilled in the art without departing from the spirit and essence of the present application, and these modifications and improvements are also considered to be within the protection scope of the present application.

Claims

1. A gesture recognition method, characterized by, The method comprises the following steps: acquiring a gesture image to be recognized; wherein the gesture image to be recognized comprises a plurality of red, green and blue (RGB) images; preprocessing the gesture image to be recognized according to a target detection algorithm, a single Gaussian model and a median filter algorithm to obtain a preprocessed gesture image to be recognized; extracting features of the preprocessed gesture image to be recognized according to an orthogonal triangular decomposition algorithm and a Tucker tensor decomposition algorithm to obtain a gesture feature set of the gesture image to be recognized; and inputting the gesture feature set of the gesture image to be recognized into a gesture recognition model to obtain a gesture recognition result; wherein the gesture recognition model is obtained according to a gesture image data set by using a transfer learning method, a full convolutional neural network and a bidirectional long short-term memory network.

2. The gesture recognition method of claim 1, wherein, The preprocessing of the gesture image to be recognized according to the target detection algorithm, the single Gaussian model and the median filter algorithm to obtain the preprocessed gesture image to be recognized comprises the following steps: tracking a gesture in the gesture image to be recognized by using the target detection algorithm, performing skin color segmentation on image information in a space position and a bounding box of the gesture in the gesture image to be recognized by using the single Gaussian model, and eliminating image information other than gesture edge and contour information in the gesture image to be recognized by using the median filter algorithm to obtain the gesture edge and contour information; and the preprocessed gesture image to be recognized comprises the gesture edge and contour information.

3. The gesture recognition method of claim 1, wherein, The feature extraction of the preprocessed gesture image to be recognized according to the orthogonal triangular decomposition algorithm and the Tucker tensor decomposition algorithm to obtain the gesture feature set of the gesture image to be recognized comprises the following steps: obtaining third-order tensor data according to an input tensor; wherein the input tensor comprises the gesture edge and contour information in the gesture image to be recognized; respectively performing orthogonal triangular decomposition on the third-order tensor data to generate three initial factor matrices; and respectively performing orthogonal iterative updating on the three initial factor matrices to obtain three updated factor matrices; wherein the gesture feature set of the gesture image to be recognized comprises the three updated factor matrices.

4. The gesture recognition method of claim 3, wherein, The process of the orthogonal iterative updating comprises the following steps: using the formula iterative updates are made; wherein, are the update results of the three factor matrices in the kth iteration, respectively; k is the iteration number; X is the input tensor; and ×1, ×2, and ×3 represent tensor matricization operations in different directions, which are used to project the tensor X into the orthogonal space of each factor matrix; are the conjugate transposes of the three factor matrices in the k-1th iteration, respectively; are the orthogonal matrices in the orthogonal triangular decomposition results of the three factor matrices, respectively; are the upper triangular matrices in the orthogonal triangular decomposition results of the three factor matrices, respectively; are the matrices obtained by parallel unfolding, are the three updated factor matrices, respectively; denotes the extraction of the first N columns from the matrix denotes the extraction of the first N columns from the matrix denotes the extraction of the first N columns from the matrix denotes the extraction of the first N columns from the matrix​​​ 5. The gesture recognition method of claim 1, wherein, The method further comprises the following steps: acquiring a gesture image data set; wherein the gesture image data set comprises a source domain data set and a target domain data set; and obtaining a gesture recognition model according to the gesture image data set by using a transfer learning method, a full convolutional neural network and a bidirectional long short-term memory network.

6. The gesture recognition method of claim 1, wherein, The obtaining of the gesture recognition model according to the gesture image data set by using the transfer learning method, the full convolutional neural network and the bidirectional long short-term memory network comprises the following steps: acquiring a source domain data set and a target domain data set; wherein the gesture image data set comprises the source domain data set and the target domain data set; preprocessing the source domain data set and the target domain data set according to a target detection algorithm, a single Gaussian model and a median filter algorithm to obtain a preprocessed source domain data set and a preprocessed target domain data set; According to the orthogonal triangular decomposition algorithm and the Tucker tensor decomposition algorithm, feature extraction is performed on the preprocessed source domain data set and the preprocessed target domain data set respectively to obtain a gesture feature set of the source domain data set and a gesture feature set of the target domain data set; The covariance matrix corresponding to the gesture feature set of the source domain data set is converted into a covariance matrix corresponding to the gesture feature set of the target domain data set to obtain a source domain data set after transfer learning; and According to the source domain data set after transfer learning, a full convolutional neural network and a bidirectional long short-term memory network are trained to obtain a gesture recognition model.

7. The gesture recognition method of claim 6, wherein, The covariance matrix corresponding to the gesture feature set of the source domain data set is converted into a covariance matrix corresponding to the gesture feature set of the target domain data set to obtain a source domain data set after transfer learning; and According to the formula and The source domain data set after transfer learning is obtained. wherein, is the feature matrix of the i-th sample in the source domain dataset after transfer learning, R TS is a mapping matrix from the gesture feature set of the target domain dataset to the gesture feature set of the source domain dataset, X Si is the feature matrix of the i-th sample in the gesture feature set of the source domain dataset, is the covariance matrix of the gesture feature set of the source domain dataset, n S is the number of samples in the gesture feature set of the source domain dataset, is the covariance matrix of the i-th gesture feature set in the gesture feature set of the source domain dataset, is the transpose matrix of ; and is the transpose matrix of X Si ; R TS T TS ; and is a normalized form of the covariance matrix of the gesture feature set of the target domain dataset, is a normalized form of the covariance matrix of the gesture feature set of the source domain dataset.

8. The gesture recognition method of claim 6, wherein, The full convolutional neural network comprises a plurality of full convolutional neural network units, each of which comprises, in sequence, a causal convolutional layer with a convolution kernel of 32@11x1, a plurality of causal expansion blocks, an activation function layer PReLU, and a plurality of multi-core blocks; wherein each of the causal expansion blocks comprises, in sequence from input to output, a dilated causal convolutional layer with a convolution kernel of 32@7x1, an activation function layer PReLU, a normalization processing layer, a dilated causal convolutional layer with a convolution kernel of 32@7x1, an activation function layer PReLU, and a normalization processing layer; each of the multi-core blocks comprises, in sequence from input to output, three parallel convolution branches, a convolutional layer with a convolution kernel of 64@1x1, an activation function layer PReLU, and a normalization processing layer; the three parallel convolution branches comprise a first convolution branch, a second convolution branch, and a third convolution branch; the first convolution branch comprises, in sequence and in series, a convolutional layer with a convolution kernel of 64@1x1 and an average pooling layer with a convolution kernel of 64@2x1; the second convolution branch comprises, in sequence and in series, a convolutional layer with a convolution kernel of 64@3x1, an average pooling layer with a convolution kernel of 64@2x1, and a convolutional layer with a convolution kernel of 64@3x3; and the third convolution branch comprises, in sequence and in series, a convolutional layer with a convolution kernel of 64@5x1, an average pooling layer with a convolution kernel of 64@2x1, and a convolutional layer with a convolution kernel of 64@5x5.

9. The gesture recognition method of claim 6, wherein, The full convolutional neural network is used to perform dimension reduction processing on the source domain data set after transfer learning, and the causal relationship between features in the source domain data set after transfer learning is extracted to obtain a fully connected feature set of the source domain data; The time features of the fully connected feature set of the source domain data are extracted using a multi-layer bidirectional long short-term memory network to obtain a time feature set of the source domain data set; The time feature set of the source domain data set is processed using a dense connection layer; and The time feature set of the source domain data set is processed using a dense connection layer; and Parameters of the full convolutional neural network and parameters of the multi-layer bidirectional long short-term memory network are optimized according to an output of the dense connection layer, to obtain a gesture recognition model.

10. The gesture recognition method of claim 1, wherein, The gesture feature set of the gesture image to be recognized is input into the gesture recognition model to obtain a gesture recognition result, and the gesture recognition result includes: In response to the gesture recognition model detecting a first preset number of continuous first standard gestures according to the gesture feature set of the gesture image to be recognized, it is determined that the first standard gesture is recognized; the first standard gesture is one gesture in a standard gesture library.

11. A gesture recognition system, characterized by The gesture recognition model includes: An image to be recognized acquisition module is configured to acquire a gesture image to be recognized; the gesture image to be recognized includes multiple red, green and blue (RGB) images; A preprocessing module is configured to perform preprocessing on the gesture image to be recognized according to a target detection algorithm, a single Gaussian model and a median filter algorithm, to obtain a preprocessed gesture image to be recognized; A feature extraction module is configured to perform feature extraction on gesture edge and contour information in the gesture image to be recognized according to an orthogonal triangular decomposition algorithm and a Tucker tensor decomposition algorithm, to obtain a gesture feature set of the gesture image to be recognized. A gesture recognition module is configured to input the gesture feature set of the gesture image to be recognized into a gesture recognition model to obtain a gesture recognition result; the gesture recognition model is obtained according to a gesture image data set by using a transfer learning method, a full convolutional neural network and a bidirectional long short-term memory network.

12. A computer device, comprising: The computer program is executed by the processor, and the processor executes the gesture recognition method according to any one of claims 1 to 10.

13. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor, and the processor executes the gesture recognition method according to any one of claims 1 to 10.

Images