Gesture tracking method in motion scene and application

Abstract

The invention relates to a gesture tracking method and application in a motion scene, and the method comprises the steps: obtaining a video stream, detecting an identification feature, and taking a video frame of which the identification feature is detected for the first time as a first frame; hand key points of the first frame are detected through a deep learning network of an image space, a standard space and a camera space, a hand three-dimensional image is reconstructed, and a state is output; for a second frame identified by the video stream, if a hand key point obtained by projecting the first frame in an area where the identification feature is located coincides with an actual hand key point of the second frame, outputting a state; for the video frame after the second frame, predicting the pose of the current video frame according to the poses of the previous two frames, and projecting the pose to the region where the identification feature is located; if the predicted hand key point coincides with the actual hand key point of the current video frame, outputting a state and repeating; the method is applied to a VR motion scene. According to the method, additional equipment is not needed, high-precision real-time gesture tracking is realized, the experience effect of a VR user is improved, the calculation amount is reduced, the speed is high, and the precision is high.

Description

Technical Field

[0001] The present invention relates to the technical field of calculation and inference, and in particular to a gesture tracking method and application in a motion scene. Background Art

[0002] With the rapid development of science and technology, technologies such as virtual reality and mixed reality are gradually becoming bridges connecting the physical world and the digital world. Gesture interaction is an important part of realizing this technology. With the continuous development of virtual reality and mixed reality technologies, the application scenarios of gesture interaction are also gradually expanding and better changing the user's interactive experience.

[0003] In the field of virtual reality, gesture interaction refers to a way of human-computer interaction through hand movements, which can enable users to operate devices more intuitively, improve user experience, and expand the interactive dimension of devices. In order to improve the feasibility of gesture interaction, it is imperative to improve the effectiveness of gesture tracking.

[0004] In the prior art, gesture tracking in virtual reality scenarios generally uses optical tracking, and there are also wearable devices that use sensors worn on the hands; the former has the advantage of low usage threshold and flexible scenarios, but its field of view is limited and the operation requirements are high. In the process of reconstructing the point cloud information of the hand through an external depth sensor or laser sensor and realizing the user's gesture tracking, the computing resources are large and the cost is high. The latter is just the opposite, but it does have more inconveniences. Summary of the invention

[0005] The present invention solves the problems existing in the prior art and provides a gesture tracking method and application in a motion scene.

[0006] The technical solution adopted by the present invention is a method for tracking gestures in a motion scene, the method comprising the following steps:

[0007] S1 obtains the video stream, detects the identification feature, and takes the video frame where the identification feature is first detected as the first frame;

[0008] S2 detects the key points of the hand using a deep learning network in image space, standard space and camera space for the first frame, reconstructs a three-dimensional image of the hand, and outputs a state; the state here includes the position of the key points of the hand;

[0009] S3, for the second frame of the video stream, if the hand key points obtained by projecting the first frame in the area where the identification feature is located coincide with the actual hand key points of the second frame, then output the status and proceed to the next step, otherwise return to step S1 or stop tracking;

[0010] S4 predicts the posture of the current video frame based on the postures of the previous two frames for any video frame after the second frame and projects it to the area where the identification features are located; if the predicted hand key points coincide with the actual hand key points of the current video frame, the state is output and S4 is repeated, otherwise it returns to step S1 or stops tracking.

[0011] Preferably, the identification feature is a palm.

[0012] Preferably, the deep learning network based on image space, standard space and camera space includes a feature output unit from image space to standard space, a feature conversion and fusion unit in standard space and a feature output unit from standard space to camera space.

[0013] Preferably, using the deep learning network to obtain the hand key points includes the following steps:

[0014] S2.1 obtaining one or more screening images based on the region where the identified feature is located in the original image of the first frame;

[0015] S2.2 inputs the screened image into the network skeleton layer and outputs one or more corresponding sets of features;

[0016] S2.3 obtaining internal parameters of the standard space based on the internal parameters and external parameters of the original image and the data of the filtered image, and inputting the internal parameters into the feature conversion layer to convert the one or more sets of features into one or more sets of standard features;

[0017] S2.4 Fusion of one or more sets of normative features;

[0018] S2.5 performs feature conversion on the fused multi-view fusion features based on the intrinsic and extrinsic parameters of the original image and the data of the filtered image to obtain the camera-based hand key points.

[0019] Preferably, in S2.1, the image area detected according to the identification features is cropped to obtain multiple image results, which are sorted based on the number of recognizable hand key points, and the two images with the largest number are screened out as screening images; in actual application, the number of recognizable hand key points that can be detected in each image is different, so the image with a larger number of recognizable hand key points is taken for more accurate feature detection.

[0020] Preferably, the network skeleton layer is RestNet or MobileNet.

[0021] Preferably, in S2.3, based on the intrinsic and extrinsic parameters of the original image and the data of the filtered image, the scale scaling intrinsic parameters are obtained, and based on the scale scaling intrinsic parameters, the scale-to-canonical space intrinsic parameters are obtained, and the scale-to-canonical space intrinsic parameters are input into the feature conversion layer.

[0022] Preferably, in S2.5, the intrinsic parameter from the canonical space to the camera is obtained by scaling the intrinsic parameter at the scale, and feature conversion is performed on the fused multi-view fusion features.

[0023] Preferably, a hand skeleton is constructed based on the hand key points obtained by the deep learning network, and after embedding the specifications, a three-dimensional image of the hand is rendered.

[0024] An application of the hand gesture tracking method in a sports scene is applied in a VR sports scene.

[0025] The present invention relates to a gesture tracking method and application in a motion scene, which obtains a video stream, detects identification features, and takes the video frame in which the identification features are detected for the first time as the first frame; for the first frame, a deep learning network of image space, standard space and camera space is used to detect hand key points, reconstruct a three-dimensional image of the hand, and output a state; for a second frame recognized by the video stream, if the hand key points obtained by projecting the first frame in an area where the identification features are located coincide with the actual hand key points of the second frame, then the state is output; for any video frame after the second frame, the posture of the current video frame is predicted based on the postures of the previous two frames and projected to the area where the identification features are located; if the predicted hand key points coincide with the actual hand key points of the current video frame, then the state is output and repeated, otherwise the state is returned or the tracking is stopped; the method is applied to VR motion scenes.

[0026] The beneficial effects of the present invention are:

[0027] (1) The grayscale camera of VR devices can be shared without adding additional equipment. The gesture tracking framework is constructed using self-supervised learning to achieve gesture tracking.

[0028] (2) Build a deep learning network that integrates image space, specification space, and camera space, and use the Lupin tracking framework to achieve high-precision real-time gesture tracking to improve the VR user experience;

[0029] (3) Cropping is performed on the perception area, and detection is performed from the image space to the standard space. Finally, the result of the camera space is generated. The result prediction replaces the call of most detection models, which reduces the amount of calculation, increases the speed and improves the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of the method of the present invention;

[0031] Figure 2 is a flow chart of the implementation of the present invention when the second frame is not lost;

[0032] Figure 3 A flow chart of obtaining the key points of the hand of the present invention;

[0033] Figure 4Schematic diagram of recognition using the palm as the identification feature of the present invention, (a) is the initialization state, and (b) is the first frame after recognition. DETAILED DESCRIPTION

[0034] The present invention is further described in detail below in conjunction with embodiments, but the protection scope of the present invention is not limited thereto.

[0035] like Figure 1 The present invention relates to a gesture tracking method in a motion scene, the method comprising the following steps:

[0036] (1) Obtaining a video stream, detecting identification features, and taking the video frame in which the identification features are first detected as the first frame;

[0037] (2) for the first frame, using a deep learning network in image space, standard space, and camera space to detect key points of the hand, reconstruct a three-dimensional image of the hand, and output the state;

[0038] (3) For the second frame of the video stream, if the hand key points obtained by projecting the first frame in the area where the identification feature is located coincide with the actual hand key points of the second frame, then the status is output and the next step is performed; otherwise, the tracking is returned to step S1 or the tracking is stopped;

[0039] (4) For any video frame after the second frame, predict the pose of the current video frame using the poses of the previous two frames and project it to the area where the identification features are located; if the predicted hand key points coincide with the actual hand key points of the current video frame, output the state and repeat (4); otherwise, return to step (1) or stop tracking.

[0040] The following is a detailed description of the steps.

[0041] (1) Obtaining a video stream, detecting identification features, and taking the video frame in which the identification features are first detected as the first frame;

[0042] The identification feature is a palm.

[0043] In the present invention, in order to ensure the effectiveness of tracking detection and reduce the difficulty of detection, the palm is used as the identification feature for detection and first frame confirmation. In the specific implementation process, it is only necessary to open the hand and display it in the specified position, and generally the detection can be achieved and tracking can be started.

[0044] (2) For the first frame, the hand key points are detected using the deep learning network in the image space, the standard space and the camera space, and the three-dimensional image of the hand is reconstructed and the state is output, such as Figure 2 As shown;

[0045] The deep learning network based on image space, standard space and camera space includes a feature output unit from image space to standard space, a feature conversion and fusion unit in standard space and a feature output unit from standard space to camera space.

[0046] In the present invention, when acquiring the first frame image, a palm detection network is used to detect the palm area in the image, and then the key points of the hand are detected through the deep learning network of the image space, the standard space and the camera space and the status is output externally; that is, firstly the perception area is obtained for the perceived identification features and preliminary processing is performed, and then the image space is detected to the standard space based on preset rules, and the output result of the camera space is generated by the rule mapping of the standard space.

[0047] The palm detection algorithm is only used in the framework when the palm is first detected or tracked. Under normal circumstances, the pose of the K-1th frame is used. k-1 And the pose of the K-2 frame k-2 Predict the pose of the Kth frame k , that is, pose k =pose k-1 +(pose k-1 -pose k-2 ), the predicted pose is projected onto the image to generate the palm area;

[0048] Specifically, if Figure 3 As shown, in combination with the embodiment, using the deep learning network to obtain the hand key points includes the following steps:

[0049] (2-1) obtaining one or more screening images based on the region where the identified features are located in the original image of the first frame;

[0050] In the present invention, after obtaining the original image of the camera of the VR device, the image area detected by the palm is cropped to crop a 96×96 perception set area, which is sorted based on the number of identifiable hand key points, and the two images with the largest number are selected as the screening images, which are image 1 and image 2;

[0051] (2-2) Input the filtered images (image 1 and image 2) into the network skeleton layer, and output the corresponding one or more sets of features, namely feature 1 and feature 2;

[0052] The network skeleton layer is RestNet or MobileNet.

[0053] (2-3) obtaining internal parameters of the standard space based on the internal parameters and external parameters of the original image and the data of the filtered image, and inputting the internal parameters into a feature conversion layer to convert the one or more sets of features into one or more sets of standard features;

[0054] In the present invention, the internal and external parameters of the original image are scaled and standardized according to the cropped 96×96 perception set area to obtain the internal parameters of the standard space, which are input to the feature conversion layer; the feature conversion layer fuses the aforementioned feature 1 and feature 2, and after inputting them to the feature conversion layer, the data of the feature space is converted to the standard space to obtain the corresponding standard feature 1 and standard feature 2;

[0055] (2-4) integrating one or more sets of normative features;

[0056] Perform multi-view fusion on standard features 1 and standard features 2 to obtain multi-view fusion features;

[0057] (2-5) Based on the internal and external parameters of the original image and the data of the filtered image, the fused multi-view fusion features are subjected to feature conversion to obtain the camera-based hand key points;

[0058] The result of the internal parameter generation of the fusion norm space to the main camera (position 0) is processed through the feature conversion layer on the multi-view fusion features to obtain the final result based on camera 0.

[0059] Furthermore, after the recognition of the key points of the hand is realized, the hand skeleton can be constructed, and then the output results can be restricted and planned based on the rules of bones and skeletons, which is conducive to rendering a three-dimensional image of the hand after embedding the specifications.

[0060] (3) For the second frame of the video stream, if the hand key points obtained by projecting the first frame in the area where the identification feature is located coincide with the actual hand key points of the second frame, then the status is output and the next step is performed; otherwise, the tracking is returned to step S1 or the tracking is stopped;

[0061] When acquiring the second frame of image, directly project the key points detected in the first frame into the second frame of image, and then detect the key points of the hand through the deep learning network of image space, standard space and camera space, and output the status externally;

[0062] Generally speaking, it takes a certain amount of time for the video to stabilize, so it is generally more convenient to confirm the first frame and the second frame during this period, so the second frame is less likely to be lost. However, if it is lost, it is necessary to reconfirm the first frame. Of course, the gesture tracking can also be ended.

[0063] (4) For any video frame after the second frame, predict the pose of the current video frame using the poses of the previous two frames and project it to the area where the identification features are located; if the predicted hand key points coincide with the actual hand key points of the current video frame, output the state and repeat (4); otherwise, return to step (1) or stop tracking.

[0064] In the present invention, if the joint point state detected in the K+1th frame image fails, it is considered that the tracking is lost, and the subsequent detection logic starts according to the 1st frame image.

[0065] In the present invention, the aforementioned hand skeleton fitting can also provide a reference for each frame output to prevent tracking errors.

[0066] The present invention also relates to an application of the gesture tracking method in a motion scene, which is applied in a VR motion scene.

[0067] In a specific application scenario, the original input image is Figure 4 As shown in (a), the hand tracking result is Figure 4 As shown in (b).

[0068] The present invention also relates to a computer-readable storage medium in application, on which a gesture tracking program in a motion scene is stored, and when the program is executed by a processor, the gesture tracking method in the motion scene is implemented.

[0069] The present invention also relates to a computer device in application, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the gesture tracking method in the above-mentioned motion scene is implemented.

[0070] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0075] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A gesture tracking method in a motion scene, characterized in that: The method comprises the following steps: S1 obtains the video stream, detects the identification feature, and takes the video frame where the identification feature is first detected as the first frame; S2 detects the key points of the hand using the deep learning network in the image space, the standard space and the camera space for the first frame, reconstructs the three-dimensional image of the hand, and outputs the state; S3: For the second frame of the video stream, if the hand key points obtained by projecting the first frame in the area where the identification feature is located coincide with the actual hand key points of the second frame, then output the status and proceed to the next step, otherwise return to step S1 or stop tracking; S4: For any video frame after the second frame, the pose of the current video frame is predicted with the pose of the previous two frames and projected to the area where the identification features are located; if the predicted hand key points coincide with the actual hand key points of the current video frame, the state is output and S4 is repeated, otherwise, return to step S1 or stop tracking.

2. The method for tracking gestures in a motion scene according to claim 1, characterized in that: The identification feature is a palm.

3. The method for tracking gestures in a motion scene according to claim 1, characterized in that: The deep learning network based on image space, standard space and camera space includes a feature output unit from image space to standard space, a feature conversion and fusion unit in standard space and a feature output unit from standard space to camera space.

4. The method for tracking gestures in a motion scene according to claim 3, characterized in that: Using the deep learning network to obtain the key points of the hand includes the following steps: S2.1 obtaining one or more screening images based on the region where the identified feature is located in the original image of the first frame; S2.2 input the screened image into the network skeleton layer and output one or more corresponding sets of features; S2.3 obtaining an internal parameter of the standard space based on the internal and external parameters of the original image and the data of the filtered image, and inputting the internal parameter into a feature conversion layer to convert the one or more sets of features into one or more sets of standard features; S2.4 Integrate one or more sets of normative features; S2.5 Based on the intrinsic and extrinsic parameters of the original image and the data of the filtered image, the fused multi-view fusion features are feature transformed to obtain the camera-based hand key points.

5. The method for tracking gestures in a motion scene according to claim 4, characterized in that: In S2.1, the image area detected according to the identification features is cropped to obtain multiple image results, which are sorted based on the number of identifiable hand key points, and the two images with the largest number are selected as screening images.

6. The method for tracking gestures in a motion scene according to claim 4, characterized in that: The network skeleton layer is RestNet or MobileNet.

7. The method for tracking gestures in a motion scene according to claim 4, characterized in that: In S2.3, the scale scaling internal parameter is obtained based on the internal and external parameters of the original image and the data of the filtered image, and the internal parameter of the scale to the canonical space is obtained based on the scale scaling internal parameter, and the internal parameter of the scale to the canonical space is input into the feature conversion layer.

8. The method for tracking gestures in a motion scene according to claim 7, characterized in that: In S2.5, the internal parameters from the canonical space to the camera are obtained by scaling the internal parameters with the scale, and feature conversion is performed on the fused multi-view fusion features.

9. The method for tracking gestures in a sports scene according to claim 3, characterized in that: A hand skeleton is constructed based on the hand key points obtained by the deep learning network, and after embedding into the specification, a three-dimensional image of the hand is rendered.

10. An application of the gesture tracking method in a motion scene according to any one of claims 1 to 9, characterized in that: Applied to VR sports scenes.

Images