A method, device and medium for hand pose estimation and recognition based on deep learning

Abstract

The application discloses a hand gesture estimation and recognition method and device based on deep learning, and a medium. The method comprises the following steps: preprocessing a target image to obtain an intermediate image; inputting the intermediate image into a preset feature extraction module to obtain first hand features and object features corresponding to the target image; performing context reasoning on the first hand features and the object features through a hand-object interaction module to enhance the first hand features and obtain second hand features; and inputting the second hand features into a multi-task joint learning module to obtain a hand gesture recognition result and a hand posture recognition result corresponding to the target image. The improved Transformer is used to enhance the hand features, thereby improving the accuracy of gesture recognition. The gesture recognition task and the hand gesture posture estimation task can be solved at the same time, and the output two-dimensional joint point heat map of the hand gesture posture estimation task is used as the input of the gesture recognition task, thereby improving the recognition ability of the network for gestures.

Description

Technical Field

[0001] This application relates to the field of visual recognition, specifically to a method, device, and medium for hand gesture estimation and recognition based on deep learning. Background Technology

[0002] Gesture recognition refers to the use of computer vision technology to extract information such as hand and body posture from images or video streams to identify a user's gesture behavior. Early gesture recognition methods mainly relied on manually extracted features such as the shape and movement of the gestures. This required manually designing feature extraction algorithms and classifying them according to rules. However, because manually designed features may not be comprehensive enough and cannot adapt to complex scenarios, the accuracy and robustness were poor.

[0003] Gesture pose estimation refers to the detection and estimation of human hand poses from images or videos using computer vision technology. This technology mainly relies on depth images or RGB images acquired from one or more sensors, and processes them through machine learning or computer vision algorithms to obtain hand pose information. Early gesture estimation methods were primarily sensor-based, which has certain limitations, such as limited accuracy and poor comfort. These sensor devices need to be in contact with or closely fitted to the human body, which may affect user comfort and natural behavior, potentially causing discomfort during prolonged use. Furthermore, sensor devices require calibration to obtain accurate pose estimation results, increasing the difficulty and time cost of use, which is not user-friendly for some ordinary users.

[0004] Gesture recognition and gesture estimation are two important problems in the field of computer vision. However, in hand interaction scenarios, since the hand is often occluded by objects, it is still very challenging to realize gesture recognition and gesture estimation using monocular RGB images. Summary of the Invention

[0005] To address the aforementioned problems, this application proposes a method, apparatus, and medium, wherein the method includes:

[0006] A target image is acquired and preprocessed to obtain an intermediate image. The intermediate image is then input into a preset feature extraction module to obtain first hand features and object features corresponding to the target image. Contextual reasoning is performed on the first hand features and object features through a hand-object interaction module to enhance the first hand features and obtain second hand features. The second hand features are then input into a multi-task joint learning module to obtain hand pose recognition results and gesture recognition results corresponding to the target image.

[0007] In one example, the preprocessing of the target image specifically includes: extracting a region of interest (ROI) image from the target image based on preset ROI features; and cropping the ROI image to obtain an intermediate image of a first preset size.

[0008] In one example, the preset feature extraction module consists of an encoder with a residual neural network and the RoiAlign algorithm; the feature extractor uses a ResNet-50 network with a residual connection structure; the step of inputting the intermediate image into the preset feature extraction module to obtain the first hand feature and object feature corresponding to the target image specifically includes: inputting the intermediate image of a first preset size into the feature extractor to obtain an intermediate feature map of a second preset size; processing the intermediate feature map using the RoiAlign algorithm to extract the feature maps of the hand and the object respectively from the intermediate feature map to obtain the first hand feature and the object feature at a third preset size.

[0009] In one example, the step of performing contextual reasoning on the first hand features and the object features through the hand-object interaction module to enhance the first hand features and obtain the second hand features specifically includes: converting the first hand features into key embeddings using a preset first parameter matrix; converting the object features into query embeddings and value embeddings using a second parameter matrix and a third parameter matrix; improving the self-attention mechanism in the Transformer model to enhance the feature representation capability of the improved Transformer model; and performing contextual reasoning on the first hand features and the object features using the improved Transformer model to enhance the first hand features and obtain the second hand features.

[0010] In one example, the improved Transformer model is used to perform contextual reasoning on the first hand features and the object features to enhance the first hand features and obtain the second hand features. Specifically, this includes: using k×k convolutions to context-encode all adjacent key embeddings within a k×k spatial grid to ensure the encoded key embeddings have contextual information, and encoding the value embeddings using 1×1 convolutions; concatenating the encoded key embeddings with the HAF detector, and then generating an attention matrix using two 1×1 convolutions and a softmax activation function; using depthwise separable convolutions to capture local features of the first hand features, and then concatenating the local features with the output values ​​of the attention module to obtain key embedding features; feeding the key embedding features into a feedforward network composed of a multilayer perceptron and layer normalization; and fusing the output of the feedforward network with the key embedding features to obtain the second hand features.

[0011] In one example, the step of inputting the second hand features into a multi-task joint learning module to obtain the hand pose recognition result and gesture recognition result corresponding to the target image specifically includes: inputting the second hand features into a two-dimensional feature point detection network and a deep regression network in the multi-task joint learning module to obtain a two-dimensional joint point heatmap and the hand pose recognition result; and inputting the two-dimensional joint point heatmap and the second hand features into a gesture recognition network to obtain the gesture recognition result.

[0012] In one example, the step of inputting the second hand features into the two-dimensional joint point localization network and the deep regression network in the multi-task joint learning module to obtain the two-dimensional joint point heatmap and the hand pose recognition result specifically includes: inputting the second hand features into the stacked hourglass network in the two-dimensional joint point localization network to determine the two-dimensional joint point heatmap; and using the difference between the predicted joint point position and the true joint point position as the loss function of the two-dimensional joint point localization network; specifically defined as follows:

[0013]

[0014] Among them, L 2D Let K represent the loss function, and p represent the number of key points. j Indicates the predicted location of the key point. The true location of the joint is represented; the hand pose parameters parameterized by the MANO model are obtained by inputting the two-dimensional joint heatmap and the second hand features into the deep regression network; the hand pose recognition result is determined by the hand pose parameters.

[0015] In one example, the step of inputting the two-dimensional joint heatmap and the second hand feature into the gesture recognition network to obtain the gesture recognition result specifically includes: combining the two-dimensional joint heatmap and the second hand feature through a 1×1 convolution and inputting it into the gesture recognition network; performing a convolution operation on the gesture recognition network on the time axis to extract dynamic features in the gesture sequence; and determining the gesture recognition network based on the dynamic features.

[0016] This application also provides a device for hand pose estimation and recognition based on deep learning, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: acquiring a target image; preprocessing the target image to obtain an intermediate image; inputting the intermediate image to a preset feature extraction module to obtain a first hand feature and an object feature corresponding to the target image; performing contextual reasoning on the first hand feature and the object feature through a hand-object interaction module to enhance the first hand feature and obtain a second hand feature; and inputting the second hand feature to a multi-task joint learning module to obtain a hand pose recognition result and a gesture recognition result corresponding to the target image.

[0017] This application also provides a non-volatile computer storage medium storing computer-executable instructions, the computer-executable instructions being configured to: acquire a target image; preprocess the target image to obtain an intermediate image; input the intermediate image to a preset feature extraction module to obtain a first hand feature and an object feature corresponding to the target image; perform contextual reasoning on the first hand feature and the object feature through a hand-object interaction module to enhance the first hand feature and obtain a second hand feature; and input the second hand feature to a multi-task joint learning module to obtain a hand pose recognition result and a gesture recognition result corresponding to the target image.

[0018] The method proposed in this application offers the following advantages: by enhancing hand features using an improved Transformer, the hand can be located more accurately, thereby improving the accuracy of gesture recognition. It can simultaneously solve both gesture recognition and gesture pose estimation tasks, performing two computer vision tasks concurrently. The output of the gesture pose estimation task, a two-dimensional joint heatmap, is used as input for the gesture recognition task, thus improving the network's ability to recognize gestures. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart illustrating a prior art gesture recognition method in an embodiment of this application;

[0021] Figure 2 This is a flowchart illustrating a deep learning-based hand pose estimation and recognition method in an embodiment of this application.

[0022] Figure 3 This is a schematic diagram illustrating the process of a deep learning-based hand pose estimation and recognition method in an embodiment of this application.

[0023] Figure 4 This is a schematic diagram of a hand-object interaction model in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of an existing Transformer model in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of a device for hand pose estimation and recognition based on deep learning, as described in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] like Figure 1 As shown, existing technologies for gesture recognition consist of three main stages: image processing, feature extraction, and feature fusion. In the image processing stage, the input image data is first cropped to obtain the hand region and the human pose region. Next, these two regions are processed through multiple 3D convolutional layers and max-pooling layers for feature extraction. Then, through fully connected layers, the hand region features and human pose region features are obtained. Finally, in the feature fusion stage, these two features are fused together to output the gesture recognition result. However, existing technologies are limited to gesture recognition and cannot complete the task of hand pose estimation. Gesture pose estimation plays a significant role in improving the experience and efficiency of gesture interaction, body interaction, and human-computer intelligent interaction. Furthermore, in hand interaction scenarios, the hand is often occluded by objects, which makes the accuracy and efficiency of gesture recognition using real-time video streams and images acquired by cameras relatively low.

[0028] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0029] Figure 2This diagram illustrates a deep learning-based hand pose estimation and recognition method provided in one or more embodiments of this specification. The method can be applied to gesture recognition and gesture pose estimation, especially when part of the hand is occluded by an object. The process can be executed by a computing device in the relevant field, and certain input parameters or intermediate results can be manually adjusted to help improve accuracy.

[0030] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0031] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0032] like Figure 2 and Figure 3 As shown in the embodiments of this application, a method for hand pose estimation and recognition based on deep learning is provided, including:

[0033] S101: Acquire the target image and preprocess the target image to obtain an intermediate image.

[0034] First, obtain images related to the hand as target images. Then, preprocess the target images to obtain intermediate images, which are used as input images for the model.

[0035] The target image can be pre-stored in the storage device of the computer device. When gesture recognition or hand posture estimation is required, the computer device can select the target image from the storage device. Of course, the computer device can also obtain the target image from other external devices. For example, the target image can be stored in the cloud. When gesture recognition or hand posture estimation is required, the computer device can obtain the target image from the cloud. This embodiment does not limit the method of obtaining the target image.

[0036] In one embodiment, during preprocessing, the region of interest (ROI) image in the target image is first extracted based on preset ROI features. Then, the ROI image is segmented to obtain an intermediate image of a first preset size, which can be 512×512×3.

[0037] S102: Input the intermediate image into the preset feature extraction module to obtain the first hand feature and object feature corresponding to the target image.

[0038] In one embodiment, the feature extraction module consists of an encoder with a residual neural network and the RoiAlign algorithm. Specifically, the feature extraction module comprises a feature extractor and a RoiAlign algorithm. The feature extractor uses a ResNet-50 network with a residual connection structure. An input image of size 512×512×3 is fed into the ResNet-50 network to obtain an intermediate feature map of a second preset size. For this intermediate feature map, the RoiAlign algorithm is used to extract feature maps of the hand and the object respectively, to obtain the first hand feature and the object feature, and the corresponding feature maps are all sampled to a fixed size of 32×32×256.

[0039] S103: The first hand feature and the object feature are subjected to contextual reasoning through the hand-object interaction module to enhance the first hand feature and obtain the second hand feature.

[0040] In hand-interaction scenarios, hands typically come into contact with objects, resulting in a high degree of similarity between hand and object poses, and numerous similarities exist in their feature maps. Therefore, contextual reasoning using information from both hand and object feature maps can enhance hand features and more accurately locate and identify hand poses. This interaction module combining hand and object feature maps can significantly improve the accuracy of gesture recognition and pose estimation, thus providing a more reliable solution for computer vision applications in hand-interaction scenarios. Therefore, by simultaneously inputting hand and object features into a Transformer-structured hand-object interaction module, enhanced hand features are obtained.

[0041] In one embodiment, the structure of the hand-object interaction module is as follows: Figure 4 As shown, we use hand features as keys and object features as queries and values. Using three parameter matrices, Wk, Wv, and Wq, we transform the hand and object feature maps into query embeddings, key embeddings, and value embeddings, respectively. Traditional Transformers use self-attention to obtain the attention matrix formed by independent queries and keys at each spatial location, but this leads to insufficient utilization of the rich contextual information between adjacent keys. Since we use hand and object features as keys and queries, respectively, we need to enhance hand features by learning hand and object contextual information. Therefore, we improve the self-attention mechanism in the Transformer to enable it to fully learn contextual information.

[0042] exist Figure 5In the traditional self-attention mechanism shown, all paired query-key relationships are learned independently without taking into account contextual information. This limits the ability of the self-attention mechanism to learn visual representations on feature maps.

[0043] To address this issue, we improved the self-attention mechanism in the Transformer, enabling it to fully utilize contextual information, enhance feature representation capabilities, and better augment hand feature maps.

[0044] like Figure 4 As shown, for the first hand feature extracted by the RoIAlign algorithm as the key, unlike the traditional self-attention mechanism which encodes each key using 1×1 convolutions, we spatially encode the key using k×k sets of convolutions for all adjacent keys within a k×k grid, thus fully utilizing the rich contextual information between adjacent keys. Simultaneously, we define object features as query and value, and encode the value using 1×1 convolutions.

[0045] K = F h W k Q = F o Wq,V=F o W v

[0046] Where K represents key embedding, and F... h As the first hand feature, W k Let F be the first parameter matrix, Q be the query embedding, and F be the first parameter matrix. o Let V be the object feature, V be the value embedding, Wq be the second parameter matrix, and W be the value embedding. v This is the third parameter matrix. The key and query, containing context information, are concatenated, and then the attention matrix is ​​generated using two 1×1 convolutions and a softmax activation function.

[0047] A = Softmax([K,Q]W α V

[0048] Where A is the output of the attention matrix, W α This is the third parameter matrix.

[0049] Meanwhile, for the first hand feature of the original input, we use depthwise separable convolution to better capture local features, and then fuse it with the output of the attention module to obtain the enhanced key feature K', which is the second hand feature.

[0050] K′=W1[DW(W1F h )]+A

[0051] Among them, W1[DW(W1F h [)] represents a local feature.

[0052] The obtained feature K' is fed into a feedforward network consisting of an MLP and a layer normalization (LN). Finally, the output of the feedforward network and feature K' are fused to obtain the enhanced hand feature map Fh'.

[0053] F′ h =K′+MLP(LN(K′))

[0054] In this way, in the hand-object interaction module, we use hand features and object features as the key and query, respectively, and enhance hand features through context encoding and attention mechanisms to more accurately locate and recognize hand poses.

[0055] S104: By inputting the second hand features into the multi-task joint learning module, the hand pose recognition result and gesture recognition result corresponding to the target image are obtained.

[0056] In one embodiment, to obtain hand gesture recognition results and hand gesture recognition results, the second hand features need to be input into the two-dimensional feature point detection network and the deep regression network in the multi-task joint learning module to obtain the two-dimensional joint heatmap and hand gesture recognition results, and the two-dimensional joint heatmap and the second hand features need to be input into the hand gesture recognition network to obtain the hand gesture recognition results.

[0057] Specifically, the gesture pose estimation task consists of two parts: a 2D feature point detection network and a depth regression network. The 2D feature point detection network uses a stacked hourglass network for hand 2D joint localization. The input is the second hand feature map output by the hand-object interaction module, and the output is a 2D heatmap for each node, with a resolution of 32×32. Through multi-level feature extraction and keypoint regression, the stacked hourglass network achieves high joint localization accuracy and exhibits good robustness, adapting well to complex situations such as hand pose, deformation, and occlusion, thus improving the stability of joint localization. Simultaneously, multi-scale feature fusion and residual connections are employed to further enhance the accuracy and stability of joint localization. The loss function of the 2D joint localization network measures the distance between the predicted joint position and the true joint position, and its definition is as follows:

[0058]

[0059] Among them, L 2D Let K represent the loss function, K represent the number of 2D joints, and p represent the number of joints. j Indicates the predicted location of the key point. This indicates the actual location of the joint.

[0060] The deep regression network consists of four CNN layers and three fully connected (FC) layers. The network input is the second hand feature map F output by the hand-object interaction module. h ′ Combined with the 2D joint heatmap output by the joint localization network, the output is the parameterized parameters of the MANO model. The parameterization method of the MANO model is achieved by decomposing the hand into two parts: a shape parameter β and a pose parameter θ. The shape parameter describes the static shape of the hand, and the pose parameter describes the dynamic posture of the hand. Using the shape parameter β and the pose parameter θ, the MANO model of the hand is defined as follows:

[0061] M(β,θ)=W(T p (β,θ),J(β),θ,ω)

[0062] Where W is the linear blend skinning (LBS) function, T p The initial pose of the hand model is given by , J represents the joint coordinates of the hand model, and ω represents the blending weights. Finally, the predicted output is calculated. The L2 distance between the ground-truth (β,θ,J,ω) and the ground-truth (β,θ,J,ω) is used as the loss function L of the deep regression network. 3D .

[0063] When performing gesture recognition, we use the output of the gesture pose estimation task: a two-dimensional joint heatmap and a second hand feature map F. h ′ We use 1×1 convolutions as input to the gesture recognition network. Since gesture recognition is a time-series classification problem, it needs to consider the temporal sequence information of hand movements. Temporal convolutions can handle the temporal relationships in the input data and capture temporal changes and dynamic features. Therefore, our designed gesture recognition network uses temporal convolutions to perform convolution operations on the time axis, which can effectively extract dynamic features from the gesture sequence. Traditional temporal convolutions convolve consecutive time frames, which may lead to the blurring or loss of key information at different time scales. Therefore, we add dilated convolutions to the temporal convolutions. Dilated convolutions can expand the receptive field while maintaining temporal resolution, enabling better capture of key information at different time scales. In the gesture recognition network, the output predicted category y is used, and we use standard classification cross-entropy as the loss function L for the gesture recognition network. g The definition is as follows:

[0064]

[0065] Where N is the number of samples, C is the number of categories, and y i,cp indicates whether the true label of the i-th sample is class c (1 or 0). i,c The probability value for predicting the i-th sample as class c.

[0066] Total Loss Function: End-to-end training is achieved using a loss function. The total loss function combines the loss functions of the 2D joint localization network, the deep regression network, and the gesture recognition network. Its specific definition is as follows:

[0067] L=λ1L 2D +λ2L 3D +λ3L g

[0068] Where L is the total loss function, and λ1, λ2, and λ3 are preset parameters.

[0069] like Figure 2 As shown in the embodiments of this application, a device for hand pose estimation and recognition based on deep learning is also provided, including:

[0070] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0071] A target image is acquired and preprocessed to obtain an intermediate image. The intermediate image is then input into a preset feature extraction module to obtain first hand features and object features corresponding to the target image. Contextual reasoning is performed on the first hand features and object features through a hand-object interaction module to enhance the first hand features and obtain second hand features. The second hand features are then input into a multi-task joint learning module to obtain hand pose recognition results and gesture recognition results corresponding to the target image.

[0072] This application embodiment also provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0073] A target image is acquired and preprocessed to obtain an intermediate image. The intermediate image is then input into a preset feature extraction module to obtain first hand features and object features corresponding to the target image. Contextual reasoning is performed on the first hand features and object features through a hand-object interaction module to enhance the first hand features and obtain second hand features. The second hand features are then input into a multi-task joint learning module to obtain hand pose recognition results and gesture recognition results corresponding to the target image.

[0074] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0075] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0076] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0081] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0082] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0083] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0084] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for hand pose estimation and recognition based on deep learning, characterized in that, include: Acquire the target image and preprocess it to obtain an intermediate image; The intermediate image is input into a preset feature extraction module to obtain the first hand feature and object feature corresponding to the target image; The first hand feature and the object feature are subjected to contextual reasoning through the hand-object interaction module to enhance the first hand feature and obtain the second hand feature. By inputting the second hand features into the multi-task joint learning module, the hand pose recognition result and gesture recognition result corresponding to the target image are obtained; The preset feature extraction module consists of an encoder with a residual neural network and the RoiAlign algorithm; the feature extraction module uses a ResNet-50 network with a residual connection structure; The step of inputting the intermediate image into a preset feature extraction module to obtain the first hand feature and object feature corresponding to the target image specifically includes: The intermediate image of the first preset size is passed to the feature extraction module to obtain an intermediate feature map of the second preset size; The RoI Align algorithm is used to process the intermediate feature map to extract the feature maps of the hand and the object respectively, so as to obtain the first hand feature and the object feature under the third preset size. The step of performing contextual reasoning on the first hand features and the object features through the hand-object interaction module to enhance the first hand features and obtain the second hand features specifically includes: The first hand feature is converted into a key embedding using a preset first parameter matrix, and the object feature is converted into a query embedding and a value embedding using a second parameter matrix and a third parameter matrix. Improve the self-attention mechanism in the Transformer model to enhance the feature representation capability of the improved Transformer model; By using the improved Transformer model, contextual reasoning is performed on the first hand features and the object features to enhance the first hand features and obtain the second hand features; The improved Transformer model is used to perform contextual reasoning on the first hand features and the object features to enhance the first hand features and obtain the second hand features. Specifically, this includes: Using k×k groups of convolutions, contextual encoding is performed on all adjacent key embeddings within a k×k grid in space, so that the encoded key embeddings have contextual information, and value embeddings are encoded using 1×1 convolutions. The encoded key embedding is concatenated with the query embedding, and then passed through two... Convolution and softmax activation functions are used to generate the attention matrix; By using depthwise separable convolution, local features of the first hand feature are captured, and then the local features are concatenated with the output value of the attention module to obtain the key embedding feature; The key embedding features are fed into a feedforward network consisting of a multilayer perceptron and a layer normalization; The output of the feedforward network and the key embedding features are fused to obtain the second hand feature.

2. The method according to claim 1, characterized in that, The preprocessing of the target image specifically includes: Based on preset features of interest, extract the region of interest (ROI) image from the target image; The image of the region of interest is cropped to obtain the intermediate image of a first preset size.

3. The method according to claim 1, characterized in that, The step of inputting the second hand features into a multi-task joint learning module to obtain the hand pose recognition result and gesture recognition result corresponding to the target image specifically includes: The second hand features are input into the two-dimensional feature point detection network and the deep regression network in the multi-task joint learning module to obtain a two-dimensional joint heatmap and the hand pose recognition result. The two-dimensional joint heatmap and the second hand features are input into the gesture recognition network to obtain the gesture recognition result.

4. The method according to claim 3, characterized in that, The step of inputting the second hand features into the two-dimensional joint point localization network and the deep regression network in the multi-task joint learning module to obtain the two-dimensional joint point heatmap and the hand pose recognition result specifically includes: The second hand feature is input into the stacked hourglass network in the two-dimensional joint localization network to determine the two-dimensional joint heatmap; the difference between the predicted joint position and the true joint position is used as the loss function of the two-dimensional joint localization network; the specific definition is as follows: in, Let K represent the loss function, and K represent the number of key points. Indicates the predicted location of the key point. Indicates the actual location of the joint; By inputting the two-dimensional joint heatmap and the second hand features into a deep regression network, hand pose parameters parameterized by the MANO model are obtained. The hand gesture recognition result is determined using the hand gesture parameters.

5. The method according to claim 3, characterized in that, The step of inputting the two-dimensional joint heatmap and the second hand features into the gesture recognition network to obtain the gesture recognition result specifically includes: The two-dimensional joint heatmap and the second hand features are combined through 1×1 convolution and then input into the gesture recognition network. The gesture recognition network is used to perform convolution operations on the time axis to extract dynamic features from the gesture sequence. The gesture recognition network is determined based on the dynamic characteristics.

6. A device for hand pose estimation and recognition based on deep learning, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the steps of the method as claimed in any one of claims 1-5.

7. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the steps of the method as described in any one of claims 1-5.

Images