Hand space feature extraction method, extraction system, extraction equipment and computer readable storage medium

Abstract

The invention relates to a hand space feature extraction method, system and device and a computer readable storage medium. The hand space feature extraction method comprises the following steps: S1, acquiring a video stream; s2, extracting a hand image based on the video stream; and S3, extracting hand spatial features based on the hand image, including inputting the hand image into the 3D feature extraction model to extract the hand spatial features, the hand spatial features including a 2D feature # imgabs0 # and a depth feature Zt. The hand space features can be effectively extracted.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent vehicle-mounted systems, and in particular to a method, system, device and computer-readable storage medium for extracting hand spatial features. Background Art

[0002] The monocular camera deep learning recognition solution mainly relies on the monocular camera to collect image data, and analyzes and processes this data through a deep learning model to achieve gesture recognition. However, this solution has some obvious problems:

[0003] First, single-frame depth estimation is unstable. In the deep learning recognition scheme of a monocular camera, depth features are usually obtained by analyzing and estimating a single-frame image. However, due to the limited information provided by a single-frame image and the influence of environmental factors, single-frame depth estimation is often not accurate and stable enough.

[0004] Secondly, the general gesture deep learning recognition model is based on the single-frame estimation results. This means that the performance of the model depends largely on the accuracy of the single-frame depth estimation. If the single-frame depth estimation is biased, the recognition results of the entire model will also be affected. Moreover, since the model is trained and optimized based on the single-frame estimation results, it may not be able to fully utilize the information on the time series, resulting in suboptimal performance when processing dynamic gestures.

[0005] The monocular camera and depth camera fusion solution attempts to combine the advantages of monocular cameras and depth cameras to improve the accuracy and robustness of gesture recognition. However, this solution also has some disadvantages:

[0006] On the one hand, the design of the fusion of monocular camera and depth camera is difficult. The fusion of two different types of cameras requires solving a series of technical problems, including camera calibration, data fusion algorithm design, system synchronization, etc. These problems increase the difficulty and complexity of system design, requiring more R&D resources and time.

[0007] On the other hand, the cost is significantly increased and the power consumption is increased. Using a depth camera usually increases the cost of the system because the price of a depth camera is relatively high. In addition, using a monocular camera and a depth camera at the same time will also increase the power consumption of the system, which is an important consideration for some mobile devices or application scenarios with high power consumption requirements. Summary of the invention

[0008] In view of the above problems in the prior art, the present invention proposes a hand space feature extraction method, extraction system, extraction device and computer-readable storage medium, which can effectively extract hand space features based on a monocular camera, thereby improving the accuracy of dynamic gesture recognition.

[0009] Specifically, the present invention proposes a method for extracting hand space features, comprising the steps of:

[0010] S1, obtain video stream;

[0011] S2, extracting a hand image based on the video stream;

[0012] S3, extracting hand spatial features based on the hand image, including:

[0013] The hand image is input into a 3D feature extraction model to extract hand spatial features, wherein the hand spatial features include 2D features and the deep feature Z t .

[0014] According to one embodiment of the present invention, the 3D feature extraction model includes an HourglassNet module, a GraphCNN module and a LFFM module. The HourglassNet module is used to extract features from the input hand image to obtain the 2D feature The GraphCNN module is used to obtain the vertices of the three-dimensional hand mesh and then obtain the depth feature Z t ,The LFFM module is used to fuse the HourglassNet module and the GraphCNN module. The LFFM module processes the output of the HourglassNet module to adapt to the input of the GraphCNN module.

[0015] According to one embodiment of the present invention, in the HourglassNet module, a lightweight SCAB operator is used to replace the Residual-Block operator to reduce computing power consumption.

[0016] According to one embodiment of the present invention, the loss function J of the 3D feature extraction model is 3D Including heat map loss J stage and depth loss J depth , the heat map loss J stage It consists of 3 stages, and the loss function J 3D The calculation formula is:

[0017]

[0018] Among them, α is the heat map loss penalty factor, and β is the multi-loss control factor.

[0019] According to one embodiment of the present invention, before inputting the hand image into a 3D feature extraction model to extract hand spatial features in step S3, the method includes:

[0020] Determine whether the hand image is valid. If not, set the hand space feature to an invalid value and output the hand space feature. If valid, input the hand image into a 3D feature extraction model to extract the hand space feature.

[0021] According to an embodiment of the present invention, after acquiring the hand spatial features in step S3, a feature filtering region is set, and features are further refined within the feature filtering region based on a spatial filtering method.

[0022] According to one embodiment of the present invention, with the current prediction position as the center, an R*R region is selected as the feature filtering region, and a Gaussian filter is used to further refine features within the feature filtering region. The two-dimensional Gaussian distribution formula is:

[0023]

[0024] Where R = 3, σ = 1.5, (x, y) is the position in the image coordinate system, μ x , μ y is the current predicted position.

[0025] According to one embodiment of the present invention, the 2D features of the hand image are obtained by using a 3D feature extraction model. and the deep feature Z t ;

[0026] 2D features Convert to relative coordinate 2D features in image coordinate system The depth feature Z t Normalized to deep feature Z t_normal .

[0027] According to one embodiment of the present invention, the depth feature The maximum distance of the target relative to the camera is Z max , the minimum distance is Z min .

[0028] According to one embodiment of the present invention, in step S2, the process of extracting the hand image includes the steps of:

[0029] S21, normalize the image to unify the pixel value scale;

[0030] S22, input the normalized image into the YOLO target detection model, and its output is subjected to the Non-Maximumsuppression algorithm to obtain N hand bounding boxes;

[0031] S23, formulating an ROI region, and filtering the hand bounding box for a target hand based on the ROI region;

[0032] S24, if the target hand exists, the filtered hand image is cropped, scaled to a resolution and normalized.

[0033] The present invention also provides a hand space feature extraction system, which is applicable to the above-mentioned hand space feature extraction method, and the extraction system comprises:

[0034] An acquisition unit, configured to acquire a video stream;

[0035] A first extraction unit, configured to extract a hand image based on the video stream;

[0036] The second extraction unit is configured to extract a hand spatial feature based on the hand image, comprising:

[0037] The hand image is input into a 3D feature extraction model to extract hand spatial features, wherein the hand spatial features include 2D features and the deep feature Z t .

[0038] The present invention also provides a device for extracting hand space features, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the aforementioned methods for extracting hand space features when executing the computer program.

[0039] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for extracting hand space features as described in any of the above items are implemented.

[0040] The present invention provides a hand space feature extraction method, extraction system, extraction device and computer-readable storage medium, which are based on a monocular camera, optimize the existing algorithm framework, enhance the algorithm feature extraction capability, effectively extract hand space features, and thus improve the accuracy of the system's dynamic gesture recognition.

[0041] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are included to provide further explanation of the present invention, and are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present invention and together with the description serve to explain the principle of the present invention.

[0043] In the attached figure:

[0044] Figure 1A flowchart of a method for extracting hand space features according to an embodiment of the present invention is shown.

[0045] Figure 2 A schematic diagram showing the distribution of joint points of a hand 21 according to an embodiment of the present invention is shown.

[0046] Figure 3 A module flow chart of hand space feature extraction according to an embodiment of the present invention is shown.

[0047] Figure 4 A test set accuracy data graph of the LFFM module according to an embodiment of the present invention is shown.

[0048] Figure 5 A schematic structural diagram of the SCAB operator in the HourglassNet module according to an embodiment of the present invention is shown.

[0049] Figure 6 A schematic diagram of the structure of a hand space feature extraction system according to an embodiment of the present invention is shown.

[0050] Figure 7 A system block diagram of a device for extracting hand space features according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0054] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the application. Meanwhile, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. The technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0055] In the description of the present application, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present application; the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0056] In addition, it should be noted that the use of words such as "first" and "second" to define components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above words have no special meaning and cannot be understood as limiting the scope of protection of this application. In addition, although the terms used in this application are selected from well-known and commonly used terms, some terms mentioned in the specification of this application may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand this application not only by the actual terms used, but also by the meaning implied by each term.

[0057] Figure 1 The flowchart of the method for extracting hand space features according to an embodiment of the present invention is shown. As shown in the figure, the method for extracting hand space features includes the following steps:

[0058] S1, obtain video stream. Obtain video stream as input data. The video stream can come from various devices, such as cameras, video files, etc., as raw data for subsequent processing. For example, in a vehicle system, the vehicle camera can capture the user's hand movements in real time and generate a video stream.

[0059] S2, extracting hand images based on the video stream. This step is used to detect whether the hand exists, separate the hand from the complex background, and generate a hand image for subsequent processing.

[0060] S3, extracting hand spatial features based on the hand image, including:

[0061] The hand image is input into the 3D feature extraction model to extract the hand spatial features, which include 2D features and the deep feature Z t The hand spatial features are closely related to the hand shape, joint positions, finger postures, etc. This step extracts the hand spatial features from the hand image through a neural network architecture.

[0062] In some examples, before step S3 inputting the hand image into the 3D feature extraction model to extract the hand spatial features, the method includes:

[0063] Determine whether the hand image is valid. If not, set the hand space feature to an invalid value and output the hand space feature. If valid, input the hand image into the 3D feature extraction model to extract the hand space feature.

[0064] It should be noted that at the beginning of the entire extraction of hand spatial features, the input hand image needs to be judged for validity. This validity judgment may be based on a variety of factors, such as image integrity, image clarity, and whether it actually contains recognizable hand content. If the hand image is found to not meet the relevant requirements after detection, it is judged to be invalid. If the hand image passes the validity test and is judged to be a valid image, the hand image is input into the 3D feature extraction model to extract hand spatial features. Hand spatial features include 2D features and depth features. 2D features mainly reflect information such as the shape and joint position of the hand on the image plane. Figure 2 The figure shows the distribution diagram of the 21 joints of the hand according to an embodiment of the present invention. As shown in the figure, the 21 joints of the hand are marked 0 to 20 respectively, the root node is the wrist joint 0, and then 4 joints are used to represent the position of each finger (thumb, index finger, middle finger, ring finger and pinky finger). 2D features It can depict the general shape and posture of the hand on a two-dimensional plane. t It reflects the distance information of each joint of the hand relative to the camera, through the depth feature Z t Combined with 2D features, it can fully describe the state of the hand in three-dimensional space, which plays a key role in subsequent tasks such as accurate recognition of dynamic gestures. and hand space features It is expressed as:

[0065]

[0066] In some examples, 2D features of the hand image are obtained by a 3D feature extraction model. and the deep feature Z t ; Then the 2D features Convert to relative coordinate 2D features in image coordinate system The depth feature Z t Normalized to deep feature Z t_normal .

[0067] It should be noted that 2D features Located in the image coordinate system, and the depth feature Located in the camera coordinate system, the two coordinate systems are different. Because the 2D features here It is only used to express the relative position relationship of the finger under a certain plane projection. It is easier to obtain it under the image coordinate system. Therefore, it is necessary to convert the 2D feature Relative coordinates in the image coordinate system Deep feature Z t Use the predefined target to normalize the distance relative to the camera, unify the information of different dimensions into a similar numerical range, and obtain the depth feature Z t_normal .

[0068] In some examples, deep features The maximum distance of the target relative to the camera is Z max , the minimum distance is Z min .

[0069] Figure 3A module flow chart of hand space feature extraction according to an embodiment of the present invention is shown. As shown in the figure, in some examples, the 3D feature extraction model includes an HourglassNet module, a GraphCNN module and a LFFM module. The HourglassNet module serves as the backbone network. The network structure is shaped like an hourglass, and repeatedly uses Top-down and Bottom-up to infer the position of the joints. The HourglassNet module is used to extract features from the input hand image and obtain 2D features. The GraphCNN module is used to obtain the vertices of the three-dimensional hand mesh, and then derive deep features from them. Specifically, it abstracts the hand into a graph structure consisting of multiple vertices and edges, where the vertices correspond to the key parts of the hand (such as joints, etc.), and the edges represent the connection relationship between these parts, and then extracts deep features from the relationship attributes such as the coordinates of the three-dimensional hand mesh vertices. The LFFM module is used to fuse the HourglassNet module and the GraphCNN module, and it processes the output of the HourglassNet module to adapt to the input of the GraphCNN module. Specifically, the LFFM module is composed of a standard MobileNet, and the SE (Squeeze and Extraction) attention mechanism is added to the last Point-wise convolution stage of each Inverted Bottleneck to enhance the channel dimension features. Secondly, the downsampling operation is replaced by Focus, which is a slicing operation on the feature map. In a feature map, a value is obtained for every other pixel, which is similar to neighboring downsampling. In this way, 4 similar and complementary new feature maps are obtained, which better preserves the integrity of the hand image features, but without information loss. In this way, the W and H information are concentrated in the channel space, and the input channel is expanded by 4 times. Finally, a 2x downsampled feature map without information loss is obtained, which provides a basis for subsequent gesture recognition. Suppose the input feature map F in ∈R C*H*W , and its final output F out ∈R 4C**2W The pseudo code is as follows:

[0070] Patch1=F in [:,∶:2,∶:2];

[0071] Patch2=F in [:,1::2,∶:2];

[0072] Patch3=F in [:,∶:2,1::2];

[0073] Patch4=F in [:,1::2,1::2];

[0074] F out =Concat(Patch1,Patch2,Patch 3, Patch4,axis=0).

[0075] Figure 4 The test set accuracy data graph of the LFFM module of an embodiment of the present invention is shown. As shown in the figure, it can be clearly observed that the curve with the LFFM module is higher than the curve without the LFFM module in most cases. This shows that under the same conditions, the LFFM module can improve the accuracy of the model. Especially in the right half of the curve, the gap between the two gradually increases, which shows that as the data processing proceeds, the advantages of the LFFM module become more obvious and can significantly improve the performance of the model.

[0076] In some examples, in the HourglassNet module, the lightweight SCAB operator is used to replace the Residual-Block operator to reduce computing power consumption. Figure 5 The structural schematic diagram of the SCAB operator in the HourglassNet module of an embodiment of the present invention is shown. As shown in the figure, the operation starts from the Channel Split module at the top, and the input feature map is split along the channel dimension. Generally, this segmentation can divide the input hand feature map into two parts, one part is used for subsequent transformation operations, and the other part is used to be directly processed by the Identity Mapping (residual connection) module. The segmented part of the feature map is first subjected to a 1x1 convolution operation, which is usually used to adjust the number of channels. This is followed by batch normalization (BN) and SiLU activation function processing, which are used to normalize and nonlinearly transform the features. The hand feature map after 1x1 convolution and activation function is then subjected to a 3x3 depth separable convolution (DWConv). Depth separable convolution decomposes the standard convolution operation into depth convolution and point-by-point convolution, which is beneficial to reduce the amount of calculation. It is then processed by batch normalization and SiLU activation function. Then, enter the SE module. The SE module is used to perform attention mechanism operations on the channel dimension. It can adaptively adjust the channel weights, enhance important channel features, and suppress unimportant channel features. The feature map output from the SE module is again subjected to a 1x1 convolution operation to further adjust the number of channels. The hand feature map that has undergone the above series of transformation operations is concatenated with another part of the hand feature map that is directly processed by Identity Mapping in Concat. This concatenation operation merges the two parts of the feature map together in the channel dimension. The concatenated hand feature map is then subjected to a 3x3 depth-separable convolution to further extract features. Finally, batch normalization and SiLU activation function processing are performed to obtain the final output hand feature map.

[0077] It is easy to understand that a 3*3 Fully Conv can be replaced by a Point-wise convolution, a 3*3 Depth-wise convolution, and a Point-wise convolution. Assume that the input hand feature map F in ∈R C*H*W , the size of the output feature map is the same as the input, and the downsampling factor of the Residual-Block operator is 0.5. Then the computing power consumption MAC (Multiply-Add Count) of the Residual-Block operator is:

[0078] MAC Res =6C 2 HW;

[0079] When the SCAB operator expansion factor t is set to 1, its computing power consumption is:

[0080] MAC SCAB =3C 2 HW+9CHW;

[0081] When C=32, the ratio of the two is:

[0082]

[0083] In this way, replacing the Residual-Block operator with the SCAB operator can reduce computing power consumption by about 45%. In addition, SCAB also introduces the concepts of Channel Splitting and Attention (attention mechanism) to further enhance feature fusion capabilities.

[0084] Furthermore, the HourglassNet module is composed of 3 Stage HourglassModules. HourglassModules usually have a unique network architecture, which includes downsampling and upsampling processes. This multi-stage HourglassModule structure helps to extract features of hand images from different levels and scales, making the extracted features more comprehensive and rich. The output calculation module of the HourglassNet module uses Residual-Block plus Point-wise convolution. In the training phase, the heat maps of all stages are involved in the training. In the inference phase, only the heat map of the last stage is output and the ArgMax operation is performed to obtain 2D features. The ArgMax operation is used to find the position with the largest response intensity in the heat map, which usually corresponds to the key feature point of the hand. In this way, the 2D features of the hand can be quickly and accurately obtained in the inference stage. The LLFM module uses the last Stage feature map of HourglassNet as input to obtain the fusion feature, and the GraphCNN module uses this as input to obtain the deep feature Z. t_normal This combination of HourglassNet, LFFM, and GraphCNN makes the entire computational process an end-to-end model, which is beneficial to model training and running speed. Let the 3D feature extraction model be M 3D , the input is the original hand image The 3D features of the hand The extraction can be characterized as:

[0085]

[0086] Among them, reference Figure 3 , only when the normalized hand image Valid, that is, only when the hand image is extracted in step S2, the normalized image will be input into the 3D feature extraction model M 3D When the hand image If invalid, skip M 3D , set the hand space feature to an invalid value.

[0087] In some examples, after obtaining the spatial features of the hand through the 3D feature extraction model in step S3, the feature filtering area is set, and the features are further refined in the feature filtering area based on the spatial filtering method. Considering that no matter how the 3D feature extraction model is optimized, the predicted position will always have a certain deviation from the actual position, and it may be unreasonable to take 3D data of a single position. Therefore, we assume that the distribution of the actual predicted position satisfies the two-dimensional Gaussian distribution. In practical applications, a method based on spatial filtering is proposed to further extract features. That is, with the current predicted position as the center, the R*R area is taken as the feature filtering area. In this example, R=3, and the feature filter uses a two-dimensional Gaussian filter. The Gaussian filter is a linear smoothing filter based on the Gaussian function, which can weightedly calculate the eigenvalues ​​in the filtering area according to the distance from the center position. The spatial features can be obtained. The two-dimensional Gaussian distribution formula is as follows:

[0088]

[0089] Among them, x, y are the positions in the image coordinate system, μ x ,μ yis the current predicted position. The variable σ determines the width of the Gaussian distribution, that is, it controls the degree of weight attenuation at different positions in the filter area. In this example, σ = 1.5.

[0090] like invalid, Assigned to That is, the values ​​of 2D features and depth features are both -1, which indicates that the key point coordinates exceed the image coordinate system and the hand is located behind the optical center of the camera, which means that the hand space feature is set to an invalid value. This assignment method can be used to identify and exclude invalid feature data in subsequent processing.

[0091] In some examples, the loss function J of the 3D feature extraction model during the training phase is 3D Including heat map loss J stage and depth loss J depth , heat map loss J stage It consists of 3 stages, each of which generates a heat map during the training phase. Heat maps are usually used to represent the response strength of hand features at different positions. Each stage's heat map has its corresponding loss, which reflects the accuracy of the model's prediction of the heat map at each stage. depth It reflects the accuracy of the model in predicting the depth features of the hand. Loss function J 3D The calculation formula is:

[0092]

[0093] Among them, α is the heat map loss penalty factor, which is used to adjust the weight of the heat map loss in the total loss. The heat map loss of different stages is adjusted by α. stage β is a multi-loss control factor, which is used to adjust the weight of depth loss in the total loss. β is used to balance the contribution of heat map loss and depth loss in the total loss function. In this example, α1=α2=0.5, α3=1.0, and β=1.3 can be set.

[0094] In some examples, during the training of the 3D feature extraction model, Batch Size represents the number of samples processed in each training iteration and is set to 128; the initial learning rate Lr 3D The step size of the model parameter update in the iteration, Lr 3DSet to 0.001; decay by 0.1 every 30 epochs of training, that is, after every 30 complete dataset traversals, the learning rate will become 0.1 times the original; a total of 100 epochs are trained, and one epoch means that the model completely traverses the dataset once. The dataset uses STB (Stereo Hand Pose Tracking Benchmark) and RHD (Rendered Hand Pose Dataset). The dataset STB is a real-world dataset containing 18,000 images, including the ground truth of 21 3D hand joint positions and the corresponding depth images. The dataset is divided into 15,000 training samples and 3,000 test samples. In order to make this setting consistent with the RHD dataset, refer to Figure 2 , moving the root joint from the center of the palm to the wrist. RHD is a synthetic dataset containing 41,258 training images and 2,728 test images. During training, the image enhancement method selects conventional random cropping, random scaling, and color domain changes. By adopting these conventional image enhancement methods in training, the diversity of training data can be effectively expanded, the robustness and generalization ability of the model can be improved, and the 3D feature extraction model can be helped to better learn the features of hand images, thereby improving the performance of subsequent gesture recognition.

[0095] In some examples, after the hand space feature extraction is implemented, the temporal hand feature is extracted based on the hand space feature. Then, the temporal hand feature is identified to obtain the gesture category and category feature. Finally, a decision is made based on the gesture category and category feature to determine the dynamic gesture. In some examples, in step S2, the process of extracting the hand image includes the steps of:

[0096] S21, normalize the image to unify the pixel value scale. Because images from different sources may have different pixel value ranges, the pixel values ​​of the image can be unified to a specific scale through normalization. Define a frame image in the video stream as I t ∈R H*W*C , that is, the height of the image at time t is H, the width is W, and the color channel is C. In this example, the video resolution is 1080P, the color encoding is RGB, and the number of frames per second is 30. Then t can be equivalent to the number of frames (1 frame sampling time is 33ms). The image is normalized, and the corresponding calculation formula is:

[0097] I t_normal =I t / 255.

[0098] S22, input the normalized image into the YOLO target detection model, and its output is subjected to the Non-Maximumsuppression algorithm to obtain N hand bounding boxes. YOLO is an efficient and widely used One-Stage target detection algorithm, which is based on a convolutional neural network and can locate and predict the category of the target object in the input image in a single forward propagation process. For hand image extraction, it can quickly detect whether there is a hand in the image and the approximate location information of the hand. The Non-Maximum suppression algorithm will filter the bounding boxes based on the confidence of each bounding box (indicating the possibility that the bounding box contains the target object) and the degree of overlap with other bounding boxes. After processing by the Non-Maximum suppression algorithm, N hand bounding boxes are obtained. These bounding boxes relatively accurately define the location of the hands in the image.

[0099] S23, formulate a ROI area (Region of Interest), and filter the target hand on the hand bounding box based on the ROI area. In order to focus on single-hand gestures and ensure that subsequent processing is targeted at a specific target hand, an ROI area needs to be formulated. This ROI area is usually a pre-set range based on specific application scenarios and requirements. It can be a specific rectangular area or other shaped area in the image, which is used to limit the hand range for subsequent processing. By calculating the intersection-and-union (IOU) of each hand bounding box and the ROI area, that is, the area of ​​the overlapping part of the hand bounding box and the ROI area divided by their combined total area, the degree of overlap between the hand bounding box and the region of interest is measured to determine the existence of the target hand. In this example, the ROI is set to the image Area. The corresponding filtering formula is:

[0100]

[0101] S24, if the target hand exists, the filtered hand image is cropped to be used as input for subsequent operation steps. To reduce subsequent computing power consumption, the cropped image is scaled to a resolution and normalized. In this example, the normalization formula for scaling to a resolution of 256*256 is:

[0102]

[0103] If the target hand does not exist, then Is set to the invalid value NULL.

[0104] The present invention also provides a hand space feature extraction system, which is applicable to the above-mentioned hand space feature extraction method. Figure 6 The schematic diagram of the structure of a hand space feature extraction system according to an embodiment of the present invention is shown. As shown in the figure, the hand space feature extraction system 600 includes:

[0105] An acquisition unit 601 is configured to acquire a video stream;

[0106] A first extraction unit 602 is configured to extract a hand image based on a video stream;

[0107] The second extraction unit 603 is configured to extract hand spatial features based on the hand image, including:

[0108] The hand image is input into the 3D feature extraction model to extract the hand spatial features, which include 2D features and the deep feature Z t .

[0109] Figure 7 The system block diagram of the hand space feature extraction device of one embodiment of the present invention is shown. As shown in the figure, the simulation device 700 may include an internal communication bus 701, a processor 702, a read-only memory (ROM) 703, a random access memory (RAM) 704 and a communication port 705. When applied on a personal computer, the simulation device 700 may also include a hard disk 706. The internal communication bus 701 can realize data communication between the components of the simulation device 700. The processor 702 can make judgments and issue prompts. In some embodiments, the processor 702 can be composed of one or more processors. The communication port 705 can realize data communication between the simulation device 700 and the outside. In some embodiments, the simulation device 700 can send and receive information and data from the network through the communication port 705. The simulation device 700 may also include different forms of program storage units and data storage units, such as a hard disk 706, a read-only memory (ROM) 703 and a random access memory (RAM) 704, which can store various data files used for computer processing and / or communication, and possible program instructions executed by the processor 702. The processor 702 executes these instructions to implement the main part of the method. The result processed by the processor 702 is transmitted to the user device through the communication port 705 and displayed on the user interface.

[0110] The above-mentioned simulation method can be implemented as a computer program, stored in the hard disk 706, and loaded into the processor 702 for execution to implement the simulation method of the present application.

[0111] The present invention also provides a device for extracting hand space features, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above-mentioned dynamic gesture recognition methods are implemented.

[0112] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned dynamic gesture recognition methods are implemented.

[0113] Among them, the specific implementation methods and technical effects of the hand space feature extraction device and computer-readable storage medium can all be referred to the embodiments of the detection method provided by the above-mentioned present invention, and will not be repeated here.

[0114] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0115] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0116] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0117] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0118] It will be apparent to those skilled in the art that various modifications and variations may be made to the above exemplary embodiments of the present invention without departing from the spirit and scope of the present invention. Therefore, it is intended that the present invention covers modifications and variations of the present invention that fall within the scope of the appended claims and their equivalent technical solutions.

Claims

1. A method for extracting hand spatial features, comprising the steps of: S1, obtain video stream; S2, extracting a hand image based on the video stream; S3, extracting hand spatial features based on the hand image, including: The hand image is input into a 3D feature extraction model to extract hand spatial features, wherein the hand spatial features include 2D features and the deep feature Z t .

2. The extraction method according to claim 1, characterized in that The 3D feature extraction model includes an HourglassNet module, a GraphCNN module and a LFFM module. The HourglassNet module is used to extract features from the input hand image to obtain the 2D features. The GraphCNN module is used to obtain the vertices of the three-dimensional hand mesh and then obtain the depth feature Z t ,The LFFM module is used to fuse the HourglassNet module and the GraphCNN module. The LFFM module processes the output of the HourglassNet module to adapt to the input of the GraphCNN module.

3. The extraction method according to claim 2, characterized in that In the HourglassNet module, the lightweight SCAB operator is used to replace the Residual-Block operator to reduce computing power consumption.

4. The extraction method according to claim 2, characterized in that The loss function J of the 3D feature extraction model 3D Including heat map loss J stage and depth loss J depth , the heat map loss J stage It consists of 3 stages, and the loss function J 3D The calculation formula is: Among them, α is the heat map loss penalty factor, and β is the multi-loss control factor.

5. The extraction method according to claim 1, characterized in that Before step S3 of inputting the hand image into a 3D feature extraction model to extract hand spatial features, the method includes: Determine whether the hand image is valid. If not, set the hand space feature to an invalid value and output the hand space feature. If valid, input the hand image into a 3D feature extraction model to extract the hand space feature.

6. The extraction method according to claim 1, characterized in that After acquiring the hand spatial features in step S3, a feature filtering region is set, and features are further refined within the feature filtering region based on a spatial filtering method.

7. The extraction method according to claim 6, characterized in that Taking the current prediction position as the center, the R*R region is selected as the feature filtering region, and a Gaussian filter is used to further refine the features in the feature filtering region. The two-dimensional Gaussian distribution formula is: Where R = 3, σ = 1.5, (x, y) is the position in the image coordinate system, μ x , μ y is the current predicted position.

8. The extraction method according to claim 1, characterized in that Obtain 2D features of hand images through 3D feature extraction model and the deep feature Z t ; 2D features Convert to relative coordinate 2D features in image coordinate system The depth feature Z t Normalized to deep feature Z t_normal .

9. The extraction method according to claim 8, characterized in that Deep Features The maximum distance of the target relative to the camera is Z max , the minimum distance is Z min .

10. The extraction method according to claim 1, characterized in that In step S2, the process of extracting the hand image includes the steps of: S21, normalize the image to unify the pixel value scale; S22, input the normalized image into the YOLO target detection model, and its output is subjected to the Non-Maximumsuppression algorithm to obtain N hand bounding boxes; S23, formulating an ROI region, and filtering the hand bounding box for a target hand based on the ROI region; S24, if the target hand exists, the filtered hand image is cropped, scaled to a resolution and normalized.

11. A hand space feature extraction system, applicable to the hand space feature extraction method according to claim 1, characterized in that: The extraction system comprises: An acquisition unit, configured to acquire a video stream; A first extraction unit, configured to extract a hand image based on the video stream; The second extraction unit is configured to extract a hand spatial feature based on the hand image, comprising: The hand image is input into a 3D feature extraction model to extract hand spatial features, wherein the hand spatial features include 2D features and the deep feature Z t .

12. A device for extracting hand space features, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for extracting hand space features as described in any one of claims 1-10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for extracting hand space features as described in any one of claims 1 to 10 are implemented.

Images