Gesture recognition detection method and system based on deep learning

Abstract

The invention discloses a gesture recognition detection method and system based on deep learning, relates to the technical field of image recognition, and establishes a good foundation for subsequent model training by collecting and preprocessing a panoramic picture of an inspection scene to generate a data set. An image segmentation model is trained on a preprocessed data set to obtain an available segmentation model, segmented gesture pictures are aligned, a training set and a verification set are divided, a gesture recognition integrated model is established, a first full connection layer of a CNN module is replaced by a GRU module to form a CGRU module, and gesture recognition is performed by means of a GPR module. According to the invention, through the whole set of process of training by using the training set and verifying by using the verification set, the gestures in the panoramic picture of the inspection scene can be effectively and accurately obtained, and the efficiency and accuracy of gesture recognition detection are improved, so that the requirements of related scenes such as inspection on gesture recognition are met.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and more specifically to a gesture recognition detection method and system based on deep learning. Background Art

[0002] At present, with the continuous development of human-computer interaction technology, gesture recognition, as a natural and intuitive interaction method, has broad application prospects in many fields, such as intelligent security monitoring, virtual reality / augmented reality interaction, smart home control, etc. Traditional gesture recognition methods are often based on manual features, such as contour features, texture features, etc., and combined with machine learning classifiers for recognition, but these methods have poor recognition accuracy and robustness in the face of complex backgrounds, diverse gesture postures, and different lighting conditions.

[0003] Deep learning technology has achieved remarkable results in the field of image recognition by virtue of its powerful ability to automatically learn features. Applying deep learning to gesture recognition detection can more effectively extract gesture features and improve the accuracy and stability of recognition. Therefore, how to provide a recognition method based on deep learning to improve the accuracy of gesture recognition is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0004] In view of this, the present invention provides a gesture recognition detection method and system based on deep learning to solve the problems existing in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A gesture recognition detection method based on deep learning, comprising:

[0007] Collect panoramic images of inspection scenes and pre-process them, including mask annotation and data enhancement, to generate data sets.

[0008] Train the image segmentation model on the preprocessed data set to obtain a trained segmentation model;

[0009] The panoramic images of the inspection scene used for gesture feature recognition are fed into the trained segmentation model, and the segmented gesture images are aligned;

[0010] Divide the aligned gesture images into training set and validation set;

[0011] Establishing a gesture recognition integrated model, in which the first fully connected layer of the CNN module is replaced by a GRU module to form a CGRU module, and the gesture features output by the CGRU module are regarded as input data to perform gesture recognition in the GPR module;

[0012] Using the training set to train the gesture recognition integrated model to obtain a trained gesture recognition integrated model;

[0013] The verification set is input into the trained gesture recognition ensemble model to obtain the gestures in the panoramic picture of the inspection scene.

[0014] Optionally, the specific method for preprocessing the collected panoramic image of the inspection scene is:

[0015] Use the annotation tool to mark the edges of the gestures in the panoramic image of the inspection scene, and then export the annotation content as a json file;

[0016] Image enhancement is performed on the annotated images using random sharpness enhancement, random color enhancement, random contrast enhancement, random brightness enhancement, and image fusion. After enhancement, one annotated image becomes five.

[0017] Optionally, the CNN module includes a hidden layer and a fully connected layer; the first hidden layer includes a convolution layer, a normalization layer and a nonlinear activation layer, a feature detector is used in each convolution layer, and data batch normalization is performed after convolution; the second hidden layer is an average pooling layer; and the fully connected layer is activated using a hyperbolic tangent function.

[0018] Optionally, the function used by the nonlinear activation layer is:

[0019] LReLU(x)={x(x≥0);αx(x<0)};

[0020] Wherein, x represents the input value, α is the parameter, and α=0.01.

[0021] Optionally, the calculation process of the GRU module is:

[0022] s t =δ(I s x t +H s m t-1 );

[0023] q t =δ(I q x t +H q m t-1 );

[0024] m t =tanm(Ix t +H(q t ⊙m t-1 ));

[0025] mt =(1-s t )⊙m t-1 +s t ⊙ m t ;

[0026] y=δ(m t I y );

[0027] Among them, I s , H s , I q , H q ,I,H,I y Represents the weight of the GRU model, I represents the weight from the input layer to the hidden layer; H is the weight of the hidden layer itself; I y represents the weight from the hidden layer to the output layer; x t is the input vector at time t, m t-1 What is saved is the information of the previous time t-1; m t is the output at time t; δ represents the S-shaped function; m t is the activation state of the hidden layer at time t; I s represents the weight from the hidden layer to the update gate, s t represents the update gate, H s is the weight from the hidden layer to the update gate at the previous time t-1; I q represents the weight from the reset gate; q t Represents the reset gate, H q represents the weight from the hidden layer to the reset gate at the previous time; y is the output of the GRU module; ⊙ represents the multiplication of the corresponding elements in the matrix.

[0028] Optionally, the specific method of training the image segmentation model on the preprocessed data set is:

[0029] Change the batch normalization layer in the model to a group normalization layer, and adjust the initialization content to skip the batch normalization initialization;

[0030] Initialize the network with pre-trained weights, use the weights trained on the COOC or PASCAL_VOC dataset, and randomly initialize the final output layer;

[0031] Each time, the preprocessed training samples and corresponding annotations are sent to the network model, and training is repeated through the two steps of forward propagation and back propagation until the maximum number of iterations is reached to minimize the loss function value.

[0032] Optionally, the specific method for aligning the segmented gesture images is:

[0033] The preprocessed image is fed into the trained segmentation model to obtain the segmentation result of the image, and the area outside the segmentation mask result in the image is set to 0, i.e. black;

[0034] Use OpenCV to calculate the segmentation edge of the segmentation result, and then use OpenCV again to calculate the minimum enclosing rectangle of the segmentation edge;

[0035] According to the four points of the minimum circumscribed rectangle, find the short side and the straight line corresponding to the short side;

[0036] Move 1 / 4 of the long side distance from the two short sides to the center of the rectangle, parallel to the long side of the rectangle, and calculate the edge area between the two short sides. The smaller area is the position of the finger, and the larger area is the position of the palm.

[0037] According to the position of the fingers and palm, rearrange the four coordinate points of the rectangle so that the fingers and palm obtained after affine transformation are definitely horizontal and upward;

[0038] The aligned original gesture image is cropped according to the rectangle size obtained by OpenCV.

[0039] A gesture recognition detection system based on deep learning, comprising:

[0040] The data collection and preprocessing module collects panoramic images of the inspection scene and preprocesses the collected panoramic images of the inspection scene, including mask annotation and data enhancement of the panoramic images of the inspection scene, to generate a data set;

[0041] The data segmentation module trains the image segmentation model on the preprocessed data set to obtain a trained segmentation model;

[0042] The data alignment module feeds the inspection scene panoramic images used for gesture feature recognition into the trained segmentation model and aligns the segmented gesture images;

[0043] The data partitioning module divides the aligned gesture images into a training set and a validation set;

[0044] A model building module is used to build a gesture recognition integrated model. In the gesture recognition integrated model, a GRU module is used to replace the first fully connected layer of the CNN module to form a CGRU module. The gesture features output by the CGRU module are regarded as input data to perform gesture recognition in the GPR module.

[0045] A model training module, which trains the gesture recognition integrated model using a training set to obtain a trained gesture recognition integrated model;

[0046] The model verification module inputs the verification set into the trained gesture recognition integrated model to obtain the gestures in the panoramic picture of the inspection scene.

[0047] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a gesture recognition detection method and system based on deep learning, which generates a data set by collecting panoramic pictures of the inspection scene and preprocessing it, laying a good foundation for subsequent model training, and then obtains a usable segmentation model by training the image segmentation model on the preprocessed data set, and then aligns the segmented gesture pictures and divides them into training sets and verification sets, and then establishes a gesture recognition integrated model, replaces the first fully connected layer of the CNN module with a GRU module to form a CGRU module and performs gesture recognition with the help of a GPR module. The present invention can effectively and accurately obtain gestures in panoramic pictures of inspection scenes by using training sets for training and verification sets to verify this whole set of processes, and improve the efficiency and accuracy of gesture recognition detection to meet the needs of gesture recognition in inspection and other related scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 A schematic diagram of the method provided by the present invention;

[0050] Figure 2 This is a schematic diagram of the system structure provided by the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The embodiment of the present invention discloses a gesture recognition detection method based on deep learning, such as Figure 1 As shown, including:

[0053] Collect panoramic images of inspection scenes and pre-process them, including mask annotation and data enhancement, to generate data sets.

[0054] Train the image segmentation model on the preprocessed data set to obtain a trained segmentation model;

[0055] The panoramic images of the inspection scene used for gesture feature recognition are fed into the trained segmentation model, and the segmented gesture images are aligned;

[0056] Divide the aligned gesture images into training set and validation set;

[0057] Establish a gesture recognition integrated model. In the gesture recognition integrated model, the GRU module is used to replace the first fully connected layer of the CNN module to form a CGRU module. The gesture features output by the CGRU module are regarded as input data for gesture recognition in the GPR module.

[0058] The gesture recognition integrated model is trained using the training set to obtain a trained gesture recognition integrated model;

[0059] The verification set is input into the trained gesture recognition ensemble model to obtain the gestures in the panoramic picture of the inspection scene.

[0060] In a specific embodiment, the specific method for preprocessing the collected panoramic image of the inspection scene is:

[0061] Use the annotation tool to mark the edges of the gestures in the panoramic image of the inspection scene, and then export the annotation content as a json file;

[0062] Image enhancement is performed on the annotated images using random sharpness enhancement, random color enhancement, random contrast enhancement, random brightness enhancement, and image fusion. After enhancement, one annotated image becomes five.

[0063] Specifically, the specific method of image data annotation is: first import the image data to be annotated into the annotation tool labelme or via, then create a polygon annotation, mark the edge of the gesture with a polygon, and mark the edge of the gesture as detailed as possible during annotation.

[0064] The specific method of data enhancement is: the enhanced content mainly includes sharpness enhancement, color enhancement, contrast enhancement, and brightness enhancement. These four enhancement methods are performed using the ImageEnhance library that PIL can only use. The enhancement method of image fusion requires first obtaining a random image for fusion, which can be downloaded online, and then scaling the background image to the size of the gesture image to be fused, and then opening the two images in RGBA mode, and randomly selecting the transparency of the background to superimpose the two.

[0065] In a specific embodiment, the CNN (convolutional neural network) module in the present invention is mainly composed of three hidden layers and two fully connected layers; the first hidden layer includes a convolution layer, a normalization layer and a nonlinear activation layer. Among them, in each of the 32 convolution layers, a feature detector with a size of 1×2 and a step size of 1 is used, and the data is batch normalized after convolution to improve the convergence speed and stability in model analysis; the second hidden layer is an average pooling layer to simplify the data dimension and prevent overfitting in the analysis. The third hidden layer has 128 convolution kernels, and a feature detector with a step size of 1×3 is used to extract more data features, and a dropout strategy layer is connected to make the model more generalizable; the hyperbolic tangent function is used to activate the two fully connected layers, and the output sequences are 1 128-length vector and 1 2-length vector respectively. The function used in the nonlinear activation layer is:

[0066] LReLU(x)={x(x≥0);αx(x<0)};

[0067] Wherein, x represents the input value, α is the parameter, and α=0.01.

[0068] Furthermore, the Gated Recurrent Unit (GRU) module: The GRU model is also used to process sequence data, and has the advantage of compensating for the gradient vanishing or exploding phenomenon that the CNN model cannot solve. The GRU module can reset the gate (q t ) and update gate (s t ) two gates to grasp the important characteristics of the data. The calculation process of the GRU module is:

[0069] s t =δ(I s x t +H s m t-1 );

[0070] q t =δ(I q x t +H q m t-1 );

[0071] m t =tanm(Ix t +H(q t ⊙m t-1 ));

[0072] m t =(1-s t )⊙m t-1 +s t ⊙ m t ;

[0073] y=δ(m t I y );

[0074] Among them, I s , H s , I q , H q ,I,H,I y Represents the weight of the GRU model, I represents the weight from the input layer to the hidden layer; H is the weight of the hidden layer itself; I y represents the weight from the hidden layer to the output layer; x t is the input vector at time t, m t-1 What is saved is the information of the previous time t-1; m t is the output at time t; δ represents the S-shaped function; m t is the activation state of the hidden layer at time t; I s represents the weight from the hidden layer to the update gate, s t represents the update gate, H s is the weight from the hidden layer to the update gate at the previous time t-1; I q represents the weight from the reset gate; q t Represents the reset gate, H q represents the weight from the hidden layer to the reset gate at the previous time; y is the output of the GRU module; ⊙ represents the multiplication of the corresponding elements in the matrix.

[0075] Further, Gaussian regression (GPR) model: When the GPR module processes N sample data sets, that is, where x i ∈R d is the input variable matrix, y n is the corresponding target value. The value between the input matrix and the output matrix is ​​as follows:

[0076] y n =f(x n )+λ n ;

[0077] Where f is the function defined in the dataset D, λ n Is subject to distribution Independent Gaussian noise. Set F = {f(x1),f(x2),…,f(x N )} have a joint Gaussian distribution:

[0078] p(F︱x1,x2,…,x N )~(0,K);

[0079] where K is the kernel matrix and k(·) is K ij= k(x i ,x j ). The squared exponential is used as the kernel function, which is defined as:

[0080]

[0081] in, is the signal equation of the kernel function; Γ is a hyperparameter symmetric matrix; is the variance of the noise, and the marginal distribution of y is as follows:

[0082]

[0083] p(y︱F,X)=∫p(y︱f,X)p(f︱X);

[0084]

[0085] Determine the hyperparameter set and establish the log-likelihood function of the GPR module as shown below:

[0086]

[0087] The optimal hyperparameters are then determined under the maximum likelihood Bayesian framework as shown in the above equation:

[0088]

[0089] Finally, the determined GPR model is used to identify new input data x' and calculate the joint distribution p(x'|X,x'), that is: Here, K' represents a (N+1)×(N+1) kernel matrix, and U represents a unit matrix.

[0090] The joint distribution following the Gaussian distribution is as follows:

[0091]

[0092] y'=[y1,y2,…,y N ,y'] T ;

[0093] k'=[k(x1,x'),…,k(x N ,x')] T ;

[0094] k'=k(x',x');

[0095] Then, the posterior distribution is determined according to Bayesian regression, and the predicted distribution of new samples x' is calculated:

[0096]

[0097] According to the above formula, the prediction interval at a 95% confidence level determined based on probability distribution theory is shown in the following equation:

[0098] [L 1-α (x'),H 1-α (x')]=[μ'-s (1-α) / 2 δ',μ'+s (1-α) / 2 δ'].

[0099] In the CGRU-GPR integrated model, the GRU module after the convolutional layer (CNN) replaces the first fully connected layer of the CNN module. The point prediction results (gesture features) of the CGRU module are regarded as input data (μ') for interval prediction in the GPR module. The gesture recognition results in the panoramic picture of the inspection scene can be obtained.

[0100] In a specific embodiment, the specific method of training the image segmentation model on the preprocessed data set is:

[0101] Change the batch normalization layer in the model to a group normalization layer, and adjust the initialization content to skip the batch normalization initialization;

[0102] Initialize the network with pre-trained weights, use the weights trained on the COOC or PASCAL_VOC dataset, and randomly initialize the final output layer;

[0103] Each time, the preprocessed training samples and corresponding annotations are sent to the network model, and training is repeated through the two steps of forward propagation and back propagation until the maximum number of iterations is reached to minimize the loss function value.

[0104] In a specific embodiment, the specific method for aligning the segmented gesture images is:

[0105] The preprocessed image is fed into the trained segmentation model to obtain the segmentation result of the image, and the area outside the segmentation mask result in the image is set to 0, i.e. black;

[0106] Use OpenCV to calculate the segmentation edge of the segmentation result, and then use OpenCV again to calculate the minimum enclosing rectangle of the segmentation edge;

[0107] According to the four points of the minimum circumscribed rectangle, find the short side and the straight line corresponding to the short side;

[0108] Move 1 / 4 of the long side distance from the two short sides to the center of the rectangle, parallel to the long side of the rectangle, and calculate the edge area between the two short sides. The smaller area is the position of the finger, and the larger area is the position of the palm.

[0109] According to the position of the fingers and palm, rearrange the four coordinate points of the rectangle so that the fingers and palm obtained after affine transformation are definitely horizontal and upward;

[0110] The aligned original gesture image is cropped according to the rectangle size obtained by OpenCV.

[0111] The specific method of alignment is: first send the gesture picture into the trained segmentation model to get the gesture segmentation result. From the gesture annotation information, we can know that the finger annotation area is relatively small and the palm annotation area is relatively large. This can be used to determine the position of the palm and fingers, and the gesture alignment can be performed based on the position of the palm and fingers. First, use findContours in OpenCV to find the edge information in the segmentation result, take the largest one, and then use minAreaRect in OpenCV to calculate the minimum enclosing rectangle of the edge. After finding the minimum enclosing rectangle, find the position of the long and short sides of the rectangle according to the four coordinate points of the rectangle. The long side must correspond to the side of the hand, and the short side must correspond to the palm or finger. Then, move 1 / 4 of the long side distance from the two short sides along the long side to the center of the rectangle to obtain two small rectangles. Use contourArea in OpenCV to calculate the area of ​​the edges of the two small rectangles. The small area represents the finger, and the large area represents the palm. In this way, the part of the hand corresponding to the four coordinate points of the rectangle can be obtained, and the four points of the rectangle are arranged in the coordinate order of the four points of the gesture after alignment, and then the gesture alignment operation is performed using affine transformation to complete the alignment. After alignment, the gesture is cropped according to the size of the rectangle.

[0112] A gesture recognition detection system based on deep learning, such as Figure 2 As shown, including:

[0113] The data collection and preprocessing module collects panoramic images of the inspection scene and preprocesses the collected panoramic images of the inspection scene, including mask annotation and data enhancement of the panoramic images of the inspection scene, to generate a data set;

[0114] The data segmentation module trains the image segmentation model on the preprocessed data set to obtain a trained segmentation model;

[0115] The data alignment module feeds the inspection scene panoramic images used for gesture feature recognition into the trained segmentation model and aligns the segmented gesture images;

[0116] The data partitioning module divides the aligned gesture images into a training set and a validation set;

[0117] Model building module, building a gesture recognition integrated model. In the gesture recognition integrated model, the GRU module is used to replace the first fully connected layer of the CNN module to form a CGRU module. The gesture features output by the CGRU module are regarded as input data to perform gesture recognition in the GPR module.

[0118] The model training module uses the training set to train the gesture recognition integrated model to obtain a trained gesture recognition integrated model;

[0119] The model verification module inputs the verification set into the trained gesture recognition integrated model to obtain the gestures in the panoramic picture of the inspection scene.

[0120] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0121] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture recognition detection method based on deep learning, characterized in that: include: Collect panoramic images of inspection scenes and pre-process them, including mask annotation and data enhancement, to generate data sets. Train the image segmentation model on the preprocessed data set to obtain a trained segmentation model; The panoramic images of the inspection scene used for gesture feature recognition are fed into the trained segmentation model, and the segmented gesture images are aligned; Divide the aligned gesture images into training set and validation set; Establishing a gesture recognition integrated model, in which the first fully connected layer of the CNN module is replaced by a GRU module to form a CGRU module, and the gesture features output by the CGRU module are regarded as input data to perform gesture recognition in the GPR module; Using the training set to train the gesture recognition integrated model to obtain a trained gesture recognition integrated model; The verification set is input into the trained gesture recognition ensemble model to obtain the gestures in the panoramic picture of the inspection scene.

2. The gesture recognition detection method based on deep learning according to claim 1, characterized in that: The specific method for preprocessing the collected panoramic images of the inspection scene is as follows: Use the annotation tool to mark the edges of the gestures in the panoramic image of the inspection scene, and then export the annotation content as a json file; Image enhancement is performed on the annotated images using random sharpness enhancement, random color enhancement, random contrast enhancement, random brightness enhancement, and image fusion. After enhancement, one annotated image becomes five.

3. The gesture recognition detection method based on deep learning according to claim 1, characterized in that: The CNN module includes a hidden layer and a fully connected layer; the first hidden layer includes a convolution layer, a normalization layer and a nonlinear activation layer, a feature detector is used in each convolution layer, and data batch normalization is performed after convolution; the second hidden layer is an average pooling layer; and the fully connected layer is activated by a hyperbolic tangent function.

4. The gesture recognition detection method based on deep learning according to claim 3 is characterized in that: The function used by the nonlinear activation layer is: LReLU(x)={x(x≥0);αx(x<0)}; Wherein, x represents the input value, α is the parameter, and α=0.

01.

5. The gesture recognition detection method based on deep learning according to claim 1, characterized in that: The calculation process of the GRU module is: s t =δ(I s x t +H s m t-1 ); q t =δ(I q x t +H q m t-1 ); m t =tanm(Ix t +H(q t ⊙m t-1 )); m t =(1-s t )⊙m t-1 +s t ⊙ m t ; y=δ(m t I y ); Among them, I s , H s , I q , H q ,I,H,I y Represents the weight of the GRU model, I represents the weight from the input layer to the hidden layer; H is the weight of the hidden layer itself; I y represents the weight from the hidden layer to the output layer; x t is the input vector at time t, m t-1 What is saved is the information of the previous time t-1; m t is the output at time t; δ represents the S-shaped function; m t is the activation state of the hidden layer at time t; I s represents the weight from the hidden layer to the update gate, s t represents the update gate, H s is the weight from the hidden layer to the update gate at the previous time t-1; I q represents the weight from the reset gate; q t Represents the reset gate, H q represents the weight from the hidden layer to the reset gate at the previous time; y is the output of the GRU module; ⊙ represents the multiplication of the corresponding elements in the matrix.

6. The gesture recognition detection method based on deep learning according to claim 1, characterized in that: The specific method of training the image segmentation model on the preprocessed data set is: Change the batch normalization layer in the model to a group normalization layer, and adjust the initialization content to skip the batch normalization initialization; Initialize the network with pre-trained weights, use the weights trained on the COOC or PASCAL_VOC dataset, and randomly initialize the final output layer; Each time, the preprocessed training samples and corresponding annotations are sent to the network model, and training is repeated through the two steps of forward propagation and back propagation until the maximum number of iterations is reached to minimize the loss function value.

7. The gesture recognition detection method based on deep learning according to claim 1, characterized in that: The specific method for aligning the segmented gesture images is as follows: The preprocessed image is fed into the trained segmentation model to obtain the segmentation result of the image, and the area outside the segmentation mask result in the image is set to 0, i.e. black; Use OpenCV to calculate the segmentation edge of the segmentation result, and then use OpenCV again to calculate the minimum enclosing rectangle of the segmentation edge; According to the four points of the minimum circumscribed rectangle, find the short side and the straight line corresponding to the short side; Move 1 / 4 of the long side distance from the two short sides to the center of the rectangle, parallel to the long side of the rectangle, and calculate the edge area between the two short sides. The smaller area is the position of the finger, and the larger area is the position of the palm. According to the position of the fingers and palm, rearrange the four coordinate points of the rectangle so that the fingers and palm obtained after affine transformation are definitely horizontal and upward; The aligned original gesture image is cropped according to the rectangle size obtained by OpenCV.

8. A gesture recognition detection system based on deep learning, characterized in that: A gesture recognition detection method based on deep learning according to any one of claims 1 to 7 is applied, comprising: The data collection and preprocessing module collects panoramic images of the inspection scene and preprocesses the collected panoramic images of the inspection scene, including mask annotation and data enhancement of the panoramic images of the inspection scene, to generate a data set; The data segmentation module trains the image segmentation model on the preprocessed data set to obtain a trained segmentation model; The data alignment module feeds the inspection scene panoramic images used for gesture feature recognition into the trained segmentation model and aligns the segmented gesture images; The data partitioning module divides the aligned gesture images into a training set and a validation set; A model building module is used to build a gesture recognition integrated model. In the gesture recognition integrated model, a GRU module is used to replace the first fully connected layer of the CNN module to form a CGRU module. The gesture features output by the CGRU module are regarded as input data to perform gesture recognition in the GPR module. A model training module, which trains the gesture recognition integrated model using a training set to obtain a trained gesture recognition integrated model; The model verification module inputs the verification set into the trained gesture recognition integrated model to obtain the gestures in the panoramic picture of the inspection scene.

Images